Electronic device with sensing assembly and method for interpreting consecutive gestures

ABSTRACT

A method for interpreting at least two consecutive gestures includes providing a sensing assembly having at least one photoreceiver and a plurality of phototransmitters, wherein each phototransmitter is positioned to emit infrared light away from the electronic device about a corresponding central transmission axis, wherein each central transmission axis is oriented in a different direction with respect to the others, and controlling emission of infrared light by each of the phototransmitters during each of a plurality of time periods. During each of the plurality of phototransmitters and for each of the plurality of time periods, a corresponding measured signal is generated which is indicative of a respective amount of infrared light which originated from that phototransmitter during that time period and was reflected by the external object prior to being received by a photoreceiver. The measured signals are evaluated to identity a first gesture, and the electronic device is controlled in response to identification of the first gesture according to a first mode of operation. A parameter of a second gesture is also determined, and the electronic device is controlled in response to the determined parameter of the second gesture according to a second mode of operation.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation-in-part of and claims the benefit ofU.S. patent application Ser. No. 12/471,062, titled “Sensing AssemblyFor Mobile Device” and filed on May 22, 2009, now U.S. Pat. No.8,304,733, which is hereby incorporated by reference herein.

STATEMENT REGARDING FEDERALLY SPONSORED RESEARCH OR DEVELOPMENT FIELD OFTHE INVENTION

The present invention relates generally to electronic devices and, moreparticularly, to an electronic device having an infrared sensingassembly for detecting predefined consecutive gestures and controllingthe electronic device.

BACKGROUND OF THE INVENTION

Mobile devices such as cellular telephones, smart phones, and otherhandheld or portable electronic devices such as personal digitalassistants (PDAs), headsets, MP3 players, etc. have become popular andubiquitous. As more and more features have been added to mobile devices,there has been an increasing desire to equip these mobile devices withinput/output mechanisms that accommodate numerous user commands and/orreact to numerous user behaviors. For example, many mobile devices arenow equipped not only with buttons or keys/keypads, but also withcapacitive touch screens by which a user, simply by touching the surfaceof the mobile device and/or moving the user's finger along the surfaceof the mobile device, is able to communicate to the mobile device avariety of messages or instructions.

It is of increasing interest that mobile devices be capable of detectingthe presence of, and determining with some accuracy the position of,physical objects located outside of the mobile devices and, moreparticularly, the presence and location of human beings (or portions oftheir bodies, such as their heads or hands) who are using the mobiledevices or otherwise are located nearby the mobile devices. By virtue ofsuch capabilities, the mobile devices are able to adjust their behaviorin a variety of manners that are appropriate given the presence (orabsence) and location of the human beings and/or other physical objects.

Although prior art devices such as capacitive touch screens are usefulas input/output devices for phones, such touch screens are fairlycomplicated electronic devices that are expensive and require a largenumber of sensing devices that are distributed in location across alarge surface area of the phone. Also, such touch screens are limitedinsofar as they only allow a user to provide input signals if the useris actually physically touching the touch screens. Further, while remotesensing devices such as infrared (or, more accurately, near-infrared)transceivers have been employed in the past in some mobile devices toallow for the detection of the presence and/or location of human beingsand/or physical objects even when not in physical contact with themobile devices, such sensing devices have been limited in variousrespects.

In particular, some such near-infrared transceivers in some such mobiledevices are only able to detect the presence or absence of a humanbeing/physical object within a certain distance from the giventransceiver (e.g., binarily detect that the human being/physical objectis within a predetermined distance or proximity to the transceiver), butnot able to detect the three-dimensional location of the humanbeing/physical object in three-dimensional space relative to thetransceiver. Also, some such transceivers in some such mobile devicesare undesirably complicated or require large numbers of components inorder to operate, which in turn renders such devices unduly expensive.

Therefore, for the above reasons, it would be advantageous if a newsensing device or sensing devices suitable for one or more types ofelectronic devices could be developed that overcame one or more of theabove-described limitations, and/or one or more other limitations.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a front elevation view of an exemplary electronic device thatemploys an exemplary pyramid-type sensing assembly capable of allowingsensing of the location of an exemplary external object (shown partiallyin cut-away), in accordance with one embodiment of the presentinvention;

FIG. 2 is a block diagram illustrating exemplary components of theelectronic device of FIG. 1;

FIG. 3 is a front perspective view showing in more detail components ofthe pyramid-type sensing assembly of FIG. 1;

FIG. 4 is a front perspective view showing components of an alternateembodiment of pyramid-type sensing assembly differing from that of FIGS.1 and 3, in accordance with another embodiment of the present invention;

FIG. 5 is a front perspective view showing components of an additionalalternate embodiment of pyramid-type sensing assembly differing fromthose of FIGS. 1, 3 and 4, in accordance with still another embodimentof the present invention;

FIG. 6 is a side elevation view of the electronic device, sensingassembly and external object (again shown partially in cutaway) of FIG.1, illustrating further the manner in which location of the externalobject is sensed;

FIG. 7 is a flow chart illustrating exemplary steps of operation of thesensing assembly (and a processing device operating in conjunctiontherewith), in accordance with at least some embodiments of the presentinvention;

FIGS. 8 and 9 are front elevation views of two exemplary electronicdevices that can employ the pyramid-type sensing assembly of FIG. 3, 4,or 5;

FIG. 10 shows a further alternate embodiment of a sensing assembly thatdiffers from that of FIG. 4 in that, instead of being a pyramid-typesensing assembly, the sensing assembly employs a lens that results inthe sensing assembly experiencing operational behavior similar to thatexperienced by pyramid-type sensing assembly of FIG. 4; and

FIG. 11 shows an additional alternate embodiment of sensing assemblydiffering from those of FIGS. 1-6 and 8-10, which includes aprism/mirror structure that receives light from a plurality of differentrespective phototransmitters positioned at respective locations apartfrom one another and apart from the location of the prism/mirrorstructure;

FIGS. 12-14 sequentially illustrate a push gesture performed by movementof a hand toward a electronic device;

FIGS. 15-17 sequentially illustrate a slide gesture performed bymovement of a hand across a electronic device;

FIG. 18 is an exemplary method for detecting a gesture;

FIG. 19 is an exemplary graph of intensities versus time for a pushgesture;

FIG. 20 is an exemplary graph of intensity versus time for a pullgesture;

FIG. 21 is an exemplary graph of intensities versus time for a slidegesture in the negative x direction;

FIG. 22 is an exemplary graph of intensities versus time for a slidegesture in the negative y direction;

FIG. 23 is a graph illustrating a horizontal slide recognition analysis;

FIG. 24 is a graph illustrating an analysis for distinguishing between ahorizontal slide and a vertical slide;

FIG. 25 is an exemplary graph of intensities versus time for a slidegesture in the positive x direction and performed with a hand in a peacesign configuration;

FIG. 26 is an exemplary graph of intensities versus time for a hoverthat occurs after a push gesture;

FIG. 27 is an exemplary graph of intensities versus time for a tiltgesture;

FIGS. 28-31 illustrate consecutive gestures including a push gesture, atilt gesture, and a slide gesture;

FIG. 32 is a graph illustrating a threshold soft blanking detectionanalysis; and

FIG. 33 is a graph illustrating a duration soft blanking'detectionanalysis.

DETAILED DESCRIPTION

An infrared sensing assembly enables detection of one or more gestures,where the gestures are predetermined patterns of movement of an externalobject relative to an electronic device that also includes a processorin communication with the sensing assembly. These gestures can bedefined to be performed in a three dimensional space and can include forexample, a push/pull gesture (movement of the object toward or away fromthe electronic device along a z axis), a slide gesture (movement of theobject in an xy plane across the electronic device), a hover gesture(stationary placement of the object for a predetermined amount of time),and a tilt gesture (rotation of the object about a roll, pitch, or yawaxis). The infrared sensing assembly can be configured in various waysand includes one or more phototransmitters which are controlled to emitinfrared light outward away from the electronic device to be reflectedby the external object, and one or more photoreceivers for receivinglight which has been emitted from the phototransmitter(s) and wasreflected from the external object.

For example, the sensing assembly can include at least one photoreceiverand multiple phototransmitters, wherein each phototransmitter ispositioned to emit infrared light away from the electronic device abouta corresponding central transmission axis, wherein each centraltransmission axis is oriented in a different direction with respect tothe others. The processor controls the phototransmitters such that eachemits infrared light at a respective portion of each of a plurality ofsequential time periods (or at the same time during each time period asfurther described below) as the external object moves in the specifiedpattern of movement. For each of the phototransmitters and for each ofthe sequential time periods, a corresponding measured signal isgenerated which is indicative of a respective amount of infrared lightwhich originated from that phototransmitter during that time period andwas reflected by the external object prior to being received by thephotoreceiver. The measured signals can be divided into measured signalsets, with each set corresponding to a respective one of thephototransmitters and including intensity values over time (overmultiple time periods). These sets can be analyzed to determinecorresponding locations of the external object at multiple points intime and to detect predetermined patterns of movement so as to identifythe gesture (including the occurrence of the gesture and its type),because each measured signal set provides information regarding whetherand when the object is in a corresponding portion of a three dimensionalspace reachable by the infrared light.

As another example, the sensing assembly can include a singlephototransmitter and multiple photoreceivers, wherein the photoreceiversare arranged so as to receive infrared light about a correspondingcentral receiving axis, wherein each central receiving axis is orientedin a different direction with respect to the others. In this case, thephototransmitter is controlled to emit light during each of a pluralityof sequential time periods, and for each of the photoreceivers and foreach of the time periods, a corresponding measured signal is generatedwhich is indicative of a respective amount of infrared light whichoriginated from the phototransmitter during that time period and wasreflected by the external object prior to being received by thatphotoreceiver. Again, the measured signals can be divided into measuredsignal sets, with each set in this case corresponding to a respectiveone of the photoreceivers and including intensity values over time (overmultiple time periods). These sets can be analyzed to determinecorresponding locations of the external object at multiple points intime and to detect predetermined patterns of movement to identify theone or more gestures.

The electronic device can be programmed to run in various differentmodes of operation, where each mode of operation links a specificidentified gesture or other determined gesture parameter to acorresponding control function of the electronic device, including insome cases a null function (where the electronic device takes noaction). The various control functions can be associated with a specificcomponent of the electronic device, such as a display, an audio devicesuch as a speaker, a camera, or one or more infrared sensors. A specificmode of operation can be activated when the electronic device is runninga particular application, when a gesture is identified, or when anotherparameter of a gesture is determined. In some cases, consecutivegestures of the same type (i.e., basic gesture type) can be associatedwith different modes of operation.

By automatically or manually switching modes, the electronic device canmore accurately interpret gestures received from a user. Also, differentmodes allow for the same gesture (or same type of gesture) to be reusedto produce different control functions for the electronic device.

Referring to FIG. 1, an exemplary electronic device 102 is shown thatincludes, among its various components, an exemplary sensing assembly104. As shown, the electronic device 102 is a mobile device such aspersonal digital assistant (PDA), albeit the electronic device is alsointended to be representative of a variety of other devices that areencompassed within the scope of the present invention including, forexample, cellular telephones, smart phones, other handheld or portableelectronic devices such as notebook or laptop computing devices,headsets, MP3 players and other portable video and audio players,navigation devices (e.g., such as those sold by Garmin International,Inc. of Olathe, Kans.), touch screen input devices, pen-based inputdevices, other mobile devices and even other devices, including a widevariety of devices that can utilize or benefit from directional controlor control based upon the sensed presence and location of one or moreexternal objects (e.g., televisions, kiosks, ATMs, vending machines,automobiles, etc.). Further included among the components of theelectronic device 102 as shown in FIG. 1 are a video screen 106, akeypad 108 having numerous keys, and a navigation key cluster (in thiscase, a “five-way navigation area”) 110.

As will be described in further detail with respect to FIG. 3, thesensing assembly 104 in the present embodiment is a first embodiment ofa pyramid-type sensing assembly that is capable of being used to detectthe presence of an object (or a collection of objects) external to theelectronic device 102 (and external to the sensing assembly itself).Depending upon the circumstance, the physical object (or objects) thatis sensed can include a variety of inanimate objects and/or, in at leastsome circumstances, one or more portions of the body of a human beingwho is using the electronic device (or otherwise is in proximity to theelectronic device) such as the human being's head or, as shown (partlyin cutaway), a hand 111 of the human being. In the present embodiment,the sensing assembly 104 not only detects the presence of such an objectin terms of whether such object is sufficiently proximate to the sensingassembly (and/or the electronic device), but also detects the object'sthree-dimensional location relative to the electronic device 102 inthree-dimensional space, and at various points over time.

In the present embodiment, the sensing assembly 104 operates bytransmitting one or more (typically multiple) infrared signals 113 outof the sensing assembly, the infrared signals 113 being generated by oneor more infrared phototransmitters (e.g., photo-light emitting diodes(photo-LEDs)). More particularly, the phototransmitters can, but neednot, be near-infrared photo-LEDs transmitting light having wavelength(s)in the range of approximately 850 to 890 nanometers. Portions of theinfrared signal(s) 113 are then reflected by an object (or more that oneobject) that is present such as the hand 111, so as to constitute one ormore reflected signals 115. The reflected signals 115 are in turn sensedby one or more infrared light sensing devices or photoreceivers (e.g.,photodiodes), which more particularly can (but need not) be suited forreceiving near-infrared light having wavelength(s) in the aforementionedrange. As will be described in further detail below, by virtue ofemploying either multiple phototransmitters or multiple photoreceivers,the three-dimensional position of the hand 111 relative to the sensingassembly (and thus relative to the electronic device) can be accuratelydetermined.

Referring to FIG. 2, a block diagram illustrates exemplary internalcomponents 200 of a mobile device implementation of the electronicdevice 102, in accordance with the present invention. The exemplaryembodiment includes wireless transceivers 202, a processor 204 (e.g., amicroprocessor, microcomputer, application-specific integrated circuit,etc.), a memory portion 206, one or more output devices 208, and one ormore input devices 210. In at least some embodiments, a user interfaceis present that comprises one or more output devices 208 and one or moreinput devices 210. The internal components 200 can further include acomponent interface 212 to provide a direct connection to auxiliarycomponents or accessories for additional or enhanced functionality. Theinternal components 200 preferably also include a power supply 214, suchas a battery, for providing power to the other internal components. Aswill be described in further detail, the internal components 200 in thepresent embodiment further include sensors 228 such as the sensingassembly 104 of FIG. 1. All of the internal components 200 can becoupled to one another, and in communication with one another, by way ofone or more internal communication links 232 (e.g., an internal bus).

Each of the wireless transceivers 202 utilizes a wireless technology forcommunication, such as, but not limited to, cellular-based communicationtechnologies such as analog communications (using AMPS), digitalcommunications (using CDMA, TDMA, GSM, iDEN, GPRS, EDGE, etc.), and nextgeneration communications (using UMTS, WCDMA, LTE, IEEE 802.16, etc.) orvariants thereof, or peer-to-peer or ad hoc communication technologiessuch as HomeRF, Bluetooth and IEEE 802.11 (a, b, g or n), or otherwireless communication technologies such as infrared technology. In thepresent embodiment, the wireless transceivers 202 include both cellulartransceivers 203 and a wireless local area network (WLAN) transceiver205, although in other embodiments only one of these types of wirelesstransceivers (and possibly neither of these types of wirelesstransceivers, and/or other types of wireless transceivers) is present.Also, the number of wireless transceivers can vary from zero to anypositive number and, in some embodiments, only one wireless transceiveris present and further, depending upon the embodiment, each wirelesstransceiver 202 can include both a receiver and a transmitter, or onlyone or the other of those devices.

Exemplary operation of the wireless transceivers 202 in conjunction withothers of the internal components 200 of the electronic device 102 cantake a variety of forms and can include, for example, operation inwhich, upon reception of wireless signals, the internal componentsdetect communication signals and the transceiver 202 demodulates thecommunication signals to recover incoming information, such as voiceand/or data, transmitted by the wireless signals. After receiving theincoming information from the transceiver 202, the processor 204 formatsthe incoming information for the one or more output devices 208.Likewise, for transmission of wireless signals, the processor 204formats outgoing information, which may or may not be activated by theinput devices 210, and conveys the outgoing information to one or moreof the wireless transceivers 202 for modulation to communicationsignals. The wireless transceiver(s) 202 convey the modulated signals toa remote device, such as a cell tower or a remote server (not shown).

Depending upon the embodiment, the input and output devices 208, 210 ofthe internal components 200 can include a variety of visual, audio,and/or mechanical outputs. For example, the output device(s) 208 caninclude a visual output device 216 such as a liquid crystal display andlight emitting diode indicator, an audio output device 218 such as aspeaker, alarm, and/or buzzer, and/or a mechanical output device 220such as a vibrating mechanism. The visual output devices 216 among otherthings can include the video screen 106 of FIG. 1. Likewise, by example,the input devices 210 can include a visual input device 222 such as anoptical sensor (for example, a camera), an audio input device 224 suchas a microphone, and a mechanical input device 226 such as a Hall effectsensor, accelerometer, keyboard, keypad, selection button, touch pad,touch screen, capacitive sensor, motion sensor, and/or switch. Themechanical input device 226 can in particular include, among otherthings, the keypad 108 and the navigation key cluster 110 of FIG. 1.Actions that can actuate one or more input devices 210 can include, butneed not be limited to, opening the electronic device, unlocking thedevice, moving the device, and operating the device.

Although the sensors 228 of the internal components 200 can in at leastsome circumstances be considered as being encompassed within inputdevices 210, given the particular significance of one or more of thesesensors 228 to the present embodiment the sensors instead are describedindependently of the input devices 210. In particular as shown, thesensors 228 can include both proximity sensors 229 and other sensors231. As will be described in further detail, the proximity sensors 229can include, among other things, one or more sensors such as the sensingassembly 104 of FIG. 1 by which the electronic device 102 is able todetect the presence of (e.g., the fact that the electronic device is insufficient proximity to) and location of one or more external objectsincluding portions of the body of a human being such as the hand 111 ofFIG. 1. By comparison, the other sensors 231 can include other types ofsensors, such as an accelerometer, a gyroscope, or any other sensor thatcan help identify a current location or orientation of the electronicdevice 102.

The memory portion 206 of the internal components 200 can encompass oneor more memory devices of any of a variety of forms (e.g., read-onlymemory, random access memory, static random access memory, dynamicrandom access memory, etc.), and can be used by the processor 204 tostore and retrieve data. The data that is stored by the memory portion206 can include, but need not be limited to, operating systems,applications, and informational data. Each operating system includesexecutable code that controls basic functions of the communicationdevice, such as interaction among the various internal components 200,communication with external devices via the wireless transceivers 202and/or the component interface 212, and storage and retrieval ofapplications and data to and from the memory portion 206. Eachapplication includes executable code that utilizes an operating systemto provide more specific functionality for the communication devices,such as file system service and handling of protected and unprotecteddata stored in the memory portion 206. Informational data isnon-executable code or information that can be referenced and/ormanipulated by an operating system or application for performingfunctions of the communication device.

Turning to FIG. 3, components of the sensing assembly 104 of FIG. 1 areshown in more detail. As shown, the sensing assembly 104 in particularincludes a pyramid-type housing structure 340 that more particularly canbe considered a tetrahedral structure that is circular in cross-sectionand has first, second, and third inclined surfaces 342, and 346,respectively, that extend downward from a triangular top surface 348.Embedded within the inclined surfaces 342, 344, and 346 are first,second and third phototransmitters 352, 354, and 356, respectively,which as noted above can be photo-LEDs suitable for emitting infraredlight. The first, second and third phototransmitters 352, 354, and 356are particularly oriented in a manner corresponding to their respectiveinclined surfaces 342, 344, and 346. That is, each of first, second, andthird center axes of transmission 362, 364, and 366 extending from therespective phototransmitters is perpendicular/normal to a respective oneof the inclined surfaces 342, 344, and 346. Further, each of the centeraxes of transmission 362, 364, and 366 is generally offset by an angle αfrom a perpendicular axis 350 extending perpendicularly/normally fromthe top surface 348. The perpendicular axis 350 in the presentembodiment is also perpendicular to the surface of the video screen 106and generally to the overall front surface of the electronic device 102upon which the sensing assembly 104, video screen 106, keypad 108, andnavigation key cluster 110 are all mounted.

Further as shown in FIG. 3, the pyramid-type sensing assembly 104 alsoincludes an additional photoelectric device in addition to thephototransmitters 352, 354, and 356 (which themselves are photoelectricdevices), namely, a photoreceiver 360 that is mounted along the topsurface 348 and, in the present embodiment, is particularly arrangedwithin the center of that surface (e.g., arranged at the center of theisosceles triangular surface). The photoreceiver 360, which as notedabove can be a photodiode or phototransistor suitable for receivinginfrared light, more particularly is arranged so that its center axis ofreception is aligned with the perpendicular axis 350. Therefore, whilethe phototransmitters 352, 354, and 356 are oriented so as to emit lightgenerally about the three center axes of transmission 362, 364, and 366,the photoreceiver 360 is oriented so as to receive light generally aboutthe perpendicular axis 350. In short, the pyramid-type sensing assembly104 can thus be described as including a single photoreceiver that issurrounded on its sides by three phototransmitters that areequally-spaced apart from one another as one proceeds around thephotoreceiver, and that are offset in terms of their vertical rotationalorientations from the vertical rotational orientation of thephotoreceiver by the same angular amount, where all of these componentsare housed within a tetrahedrally-shaped housing with surfaces thatcorrespond to the rotational orientations of the phototransmitters andphotoreceiver. In other cases, both multiple phototransmitters andmultiple photoreceivers can be used, for example, with thephototransmitters oriented as described above, and such that one or moreof the photoreceivers are oriented to better receive reflected lightthat emitted from a respective phototransmitter.

Due to the particular orientations of the phototransmitters 352, 354,356 and the photoreceiver 360, light from the respectivephototransmitters is directed generally in three different directionscorresponding to the center axes of transmission 362, 364, 366 (althoughthere may be some overlapping of the ranges within which the respectivephototransmitters direct light), while the photoreceiver 360 due to itscentral location and orientation along the perpendicular axis 350 ispotentially capable of receiving reflected light from a variety ofdirections that can overlap the directions of transmission of each ofthe three of the phototransmitters. More particularly, because thephotoreceiver 360 is capable of receiving light from a wider range ofangles about the perpendicular axis 350 than the ranges of angles aboutthe respective center axes of transmission 362, 364, 366 within whichthe respective phototransmitters are capable of directing light, in thepresent embodiment the overall sensing assembly 104 operates predicatedupon the assumption that the photoreceiver is capable of receiving lightthat is reflected off of an object such as the hand 111 even though thereflected light may have originated from any one or more of the threephototransmitters.

Further as illustrated in FIG. 3, the components of the sensing assembly104 described above can be mounted directly upon a circuit board 368upon which other components such as components 369 are mounted. Byvirtue of this direct mounting of the sensing assembly 104, the sensingassembly 104 need not protrude out far from the overall surface of theelectronic device 102 on which the video screen 106, keypad 108 andnavigation key cluster 110 are all situated. In the embodiment of FIG.3, the sensing assembly 104 is particularly shown to be implemented neara top edge of the front surface of the electronic device 102, whichoften is the location of a speaker of a mobile phone. However, asdiscussed further below, other positions for such a sensing assembly arealso possible.

Turning next to FIG. 4, the present invention is intended to encompassnumerous other pyramid-type sensing assemblies other than that shown inFIG. 3. For example, as shown in FIG. 4, a sensing assembly 400 isemployed that has a more conventional four-sided pyramid-type shape (bycomparison with the tetrahedral shape of FIG. 3). More particularly, thesensing assembly 400 has a pyramid-type housing structure 471 havingfour edges forming a square perimeter 472, and four inclined surfaces474, 476, 478, and 480. Similar to the sensing assembly 104 of FIG. 3,the housing structure 471 of the sensing assembly 400 additionallyincludes a top surface 482 from which each of the respective fourinclined surfaces 474, 476, 478, and 480 slope downwardly. With respectto the sensing assembly 104, phototransmitters 484, 486, 488, and 490,such as photo-LEDs, are each situated along a respective one of theinclined surfaces 474, 476, 478, and 480, and a photoreceiver 492, suchas a photodiode, is mounted on the top surface 482. Thus, similar to thesensing assembly 104, the sensing assembly 400 includes multiplephototransmitters arranged about (and equally spaced about) a singlephotoreceiver that is centrally positioned in between thephototransmitters.

Further as shown in FIG. 4, a center axis of reception of thephotoreceiver 492 again is aligned with a perpendicular axis 493normally extending from the top surface 482, which is angularly spacedapart by an angle β from each first, second, third, and fourth centeraxes of transmission 494, 496, 498, and 499 of the respectivephototransmitters 484, 486, 488, and 490. In other embodiments, one ormore of the phototransmitters can be arranged so as to have anassociated angle different than the others. Thus, as with the sensingassembly 104, the respective phototransmitters 484, 486, 488, 490 eachare vertically rotationally offset relative to the perpendicular axis493 (and thus relative to the center axis of reception of thephotoreceiver 492) in a manner corresponding to the slopes of therespective inclined surfaces 474, 476, 478, 480 with which thephototransmitters are associated. Also as with the sensing assembly 104,the photoreceiver 492 is capable of receiving light within a much widerrange of angles relative to the perpendicular axis 493 than therespective phototransmitters 484, 486, 488, 490 transmit light relativeto their respective center axes of transmission 494, 496, 498, 499, andoperation of the sensing assembly 400 again is predicated upon theassumption that the photoreceiver 492 is capable of receiving light thatis reflected off of an external object that may have been transmitted byany one or more of the phototransmitters 484, 486, 488, 490.

Referring next to FIG. 5, a further alternate embodiment of a sensingassembly 500 is shown. In this embodiment, the sensing assembly 500again has a pyramid-type housing structure 501 with four inclinedsurfaces 502, 504, 506 and 508, respectively, each of which is inclinedand slopes downwardly from a horizontal top surface 510. In thisembodiment, however, the sensing assembly 500 does not employphototransmitters on the inclined surfaces 502, 504, 506 and 508, butrather has mounted on those surfaces first, second, third and fourthphotoreceivers 512, 514, 516, and 518, respectively. Further, instead ofemploying a photoreceiver along the top surface 510, instead aphototransmitter 520 is mounted along (or, more particularly, recessedwithin) that surface. Given this design, in contrast to the embodimentsof FIGS. 3 and 4, it is expected that light emitted from thephototransmitter 520, upon being reflected by an object or objectsexternal to the electronic device (e.g., the hand 111), will bereflected to one or more of the photoreceivers 512, 514, 516 and 518.

Although not shown in FIGS. 3-5, in some circumstances thephotoreceivers 360, 492 and 512, 514, 516, 518 need not extend up to thevery outer surfaces of the sensing assemblies/pyramid-type housingstructures, but rather above those photoreceivers additional structurescan be positioned, such as transparent windows or walls that provideprotection for the photoreceivers and/or provide additional desiredoptical properties. In some such circumstances, for example, suchtransparent windows can constitute waveguides (or “V-notches” orCompound Parabolic Concentrator (CPC) waveguides) that serve to betterdirect incoming reflected light into the photoreceivers, and/or thatserve as lenses for magnification purposes, improving gain and/orminimizing local coupling. In some cases, certain portions of thesurfaces surrounding the photoreceivers can be coated with silver orcopper paint (or other shiny material) so as to reflect infrared lighttoward the photoreceivers. Also, in some cases, the photoreceiversthemselves can be shielded (e.g., electrically shielded) or can be“black diodes” to alleviate background lighting issues, internalreflection/noise and/or noise from the phototransmitters of the sensingassembly. These types of features can be of particular interest inrelation to the embodiments such as those of FIGS. 3-4 involving asingle photoreceiver.

Further, depending upon the embodiment, the photoreceivers can take avariety of forms including, for example, angle-diversity receivers orfly-eye receivers. Depending upon the embodiment, various filters can beemployed above the photoreceivers and/or phototransmitters to filter outundesired light. Different filters can in some circumstances be employedwith different ones of the phototransmitters/photoreceivers, forexample, to allow for different colors of light to be associated with,transmitted by, or received by, the different components.

Each of the embodiments of sensing assemblies shown in FIGS. 3, 4 and 5are similar (notwithstanding their differences) in that multiplephototransmitters and/or photoreceivers are co-located (that is,commonly located) in a single or shared small region, that is, a regionthat is small by comparison with the overall surface dimensions of theelectronic device on which the sensing assemblies are intended to beimplemented. Further, in at least these embodiments, it is additionallythe case that either only one photoreceiver (where multiplephototransmitters are present) or only one phototransmitter (wheremultiple photoreceivers are present) is used, although the presentinvention is also intended to encompass other embodiments in which thereare multiple phototransmitters as well as multiple photoreceivers thatare co-located. Also, as already mentioned with respect to FIG. 3, ineach of these embodiments, the phototransmitter(s)/photoreceiver(s) andassociated pyramid-type housing structures can be (but need not be)mounted on a circuit board along with other circuit components.

The co-location of the phototransmitter(s)/photoreceiver(s) mounted inthe pyramid-type housing structures in accordance with embodiments suchas those of FIGS. 3-5 is beneficial in several regards. First, by virtueof the co-location of photoreceiving and phototransmitting devices inthe manners shown, including the particular orientations shown (e.g.,relative to the perpendicular axes 350, 493), it is possible for therespective sensing assembly to allow for the sensing not only of thepresence of an external object (that is, to detect the fact that theobject is within a given distance or proximity relative to the sensingassembly) but also the location of an external object such as the hand111 in three-dimensional space relative to the sensing assembly. Indeed,this can be accomplished even though, in each of the embodiments ofFIGS. 3-5, there is only one of either a phototransmitter or aphotoreceiver, as discussed in further detail with reference to FIG. 6below. Further, by virtue of the co-location of the photoreceiving andphototransmitting devices in the manners shown, in the pyramid-typehousing structures, the resulting sensing assemblies are both robust andconcentrated (rather than distributed) in design. Thus, the sensingassemblies can potentially be discrete structures that can beimplemented in relation to many different types of existing electronicdevices, by way of a relatively simple installation process, as add-onor even after-market devices.

It should be noted that the particular angular ranges associated withthe transmission or reception of light by the differentphototransmitters and photoreceivers associated with sensing assembliessuch as those described above can vary with the embodiment and dependingupon the intended purpose. As noted earlier, typically photoreceiverscan have a range of reception (e.g., very broad such as a 60 degreerange to narrow based on an associated integrated lensing scheme) thatis larger than the range of transmission of the phototransmitters (e.g.,a 20 degree range). Nevertheless, this need not be the case in allembodiments. That said, it should further be noted that it isanticipated that, in practical implementations, the embodiments of FIGS.3 and 4 may be superior to that of FIG. 5 insofar as it is commonly thecase that the angular range over which a given photoreceiver is capableof receiving light is considerably larger than the angular range overwhich a phototransmitter is capable of sending light and as such moresevere tilting of the photoreceivers in the embodiment of FIG. 5 wouldbe need to distinguish between reflected light signals. Also, the use ofa single photoreceiver to receive the reflected light originating frommultiple phototransmitters as is the case with the embodiments of FIGS.3-4 typically allows for simpler sensing circuitry to be used becausereceiver circuitry is usually more complex than transmitting circuitry.

Turning to FIG. 6, a side-view of the electronic device 102 and hand 111of FIG. 1 is provided (with the hand again shown partly in cutaway) tofurther illustrate how the sensing assembly 104 with its co-locatedphototransmitters and single photoreceiver is capable of detecting thepresence and location of the hand (or a portion thereof, e.g., afinger). As illustrated, when the hand 111 is present and positionedsufficiently proximate the sensing assembly 104, it is often if nottypically (or always) the case that the hand will be positioned at alocation that is within the range of transmission of light of at leasttwo if not all three of the phototransmitters 352, 354 and 356 of thesensing assembly 104. In the present example, therefore, when light istransmitted from more than one of the phototransmitters, for example,the phototransmitters 352 and 354 as shown, emitted light 672 and 674from the respective phototransmitters reaches the hand at an angle andis reflected off of the hand so as to generate corresponding amounts ofreflected light 676 and 678, respectively. Given the position of thephotoreceiver 360 in between the phototransmitters 352, 354, theseamounts of reflected light 676, 678 both reach the photoreceiver and aresensed by the photoreceiver as shown.

Referring additionally to FIG. 7, a flow chart is provided that shows inmore detail one exemplary manner of operating the components of thesensing assembly 104 so as to determine the location of an externalobject (e.g., the hand 111), and in which the phototransmitters are eachcontrolled to emit light during each of one or more sequential timeperiods. More specifically with respect to FIG. 7, after startingoperation at a step 780, a first of the phototransmitters of the sensingassembly 104 (e.g., the phototransmitter 352) is selected at a step 782.Then at a step 784, the selected phototransmitter is actuated so thatinfrared light is emitted from that phototransmitter. That light canthen proceed towards the external object (e.g., as the emitted light 672of FIG. 6) and, upon reaching the external object, some of that light isreflected by the external object (e.g., as the reflected light 676). Ata step 786 that reflected light is in turn received by the photoreceiver(e.g., the photoreceiver 360) and the photoreceiver correspondinglysends a signal to a processing device (and/or memory device) thatrecords the received information. At a step 788 it is further determinedwhether all of the phototransmitters have been actuated. If this is notthe case, then another of the remaining phototransmitters (e.g., thephototransmitter 354) is selected at a step 790 and then the steps 784,786, and 788 are repeated (e.g., such that the emitted light 674 istransmitted and the reflected light 678 is received by thephotoreceiver). If however at the step 788 it is determined that all ofthe phototransmitters have been actuated and, consequently, reflectedlight signals have been received by the photoreceiver in relation to thelight emitted by each of those phototransmitters during a correspondingtime period, then at a step 792 the information from the photoreceiveris processed to determine the location of the external object in threedimensional space.

The signal information from the photoreceiver can be processed todetermine the location of the external object as follows. The exemplarymanner of operation described in FIG. 7 effectively constitutes a formof time division multiplexing in which the various phototransmitters areturned on and off one at a time in a serial manner, such that there aresuccessive time windows or respective portions of each time periodassociated with the actuation of the different phototransmitters. Giventhat the external object being sensed is positioned relatively close tothe transmitters and photoreceiver, these successive time windows notonly constitute the respective windows within which the differentphototransmitters are actuated but also constitute the respectivewindows within which light originating at the respectivephototransmitters is emitted, reflected off of an external object, andreceived at the photoreceiver. Thus, the signals provided from thephotoreceiver that are indicative of the intensity/amount of lightreceived by the photoreceiver during any given time window can becompared relative to the intensity/amount of light given off by thephototransmitter known to have emitted light during that time window,and such comparisons can serve as a measurement of the proportion oflight emitted by a given phototransmitter that actually returns to thephotoreceiver due to reflection by the external object. Suchmeasurements in turn serve as indications of the proximity of theexternal object to the respective phototransmitters and photoreceiverbetween which the light is communicated.

Thus, in FIG. 7, the phototransmitters are controlled such that each oneemits light during a respective, non-overlapping portion of each of oneor more time periods, and the photoreceiver detects measured signals,each of which can be associated with a corresponding one of thephototransmitters based on timing. However, in other cases, thephototransmitters can emit light at different frequencies (wavelengths)or bandwidths and perhaps different colors such that thephototransmitters can be controlled to each emit light at the same timeduring each of one or more sequential time periods. In this case,receiver circuitry can be provided so as to electronically filter themeasured signals by frequency such that each measured signal can beassociated with a respective one of the phototransmitters. Another wayto differentiate the measured signals when the sensing assembly usesdifferent colors of light emitted by the phototransmitters involves theuse of an optical filter which can separate the different colorwavelengths of light, with the corresponding use of a matchedphotoreceiver for each of the colors.

In any case, for such measurements to be more accurate, moreparticularly, certain additional information can be taken into account,or at least one or more assumptions can be made. For example, suchmeasurements particularly become more accurate as an indication ofproximity if one can make an accurate assumption regarding the physicalreflectivity of the external object, something which is typicallypossible to a sufficiently high degree in practice. Additionalconsiderations to take into account can include surface texture, size,shape, consistency, material, object orientation/direction. Predictingabsolute reflection levels can be challenging in such environments andcan require a calibration procedure. Also, it may be desirable to relyon other technologies which are inherently less susceptible to abovefactors (such as ultrasonic sensing) to more accurately measure objectrange and feed that information back to the processor to optimize thesensing assembly performance and improve tracking capabilities.Additionally, the physical positions/orientations of thephototransmitters and photoreceivers also influence the measurements andshould be taken into account. Further, angular variations in thetransmission and reception of the phototransmitters and photoreceiveralso should be taken into account. In this respect, and as alreadydiscussed, each of the phototransmitters has a respective center axis oftransmission and the photoreceiver similarly has a respective centeraxis of reception. The transmission intensity from the phototransmitterschanges (typically decreases) as the angle between that center axis oftransmission and the actual direction of transmission increases, andlikewise the reception ability of the photoreceiver also changes(typically decreases) as the angle between the center axis of receptionand the actual direction of reception increases. Typically, the degreesto which these quantities vary as one moves away from the center axes oftransmission or reception are known properties associated with thephototransmitters and photoreceivers.

Assuming then that a processing device has all of these types ofinformation or at least can rely upon reasonable assumptions concerningthese issues, the processing device receiving the signals from thephotoreceiver (e.g., the processor 204 of FIG. 2, which also can controlactuation of the phototransmitters) is not only able to determine thedistance of the external object from the infrared sensing assembly, butmore particularly is also able to determine the three-dimensionallocation of the external object by a type of triangulation calculation(or calculations). More particularly, after the processing device hasassociated the multiple amplitude (intensity) levels indicated by thephotoreceiver as occurring during the different time windows withinwhich multiple phototransmitters have respectively been actuated totransmit light, the processing device can not only determine theamount/intensity of infrared light emanating from each phototransmitterthat is reflected back to the photoreceiver but also can compare therelative amounts/intensities of infrared light originating at thedifferent phototransmitters that are reflected back to thephotoreceiver, so as to determine the location of the external objectrelative to the infrared sensing assembly. Generally speaking, as theamounts/intensities of infrared light reflected back to thephotoreceiver tend to differ from one another based upon thephototransmitter from which the infrared light originated, this tends toindicate that the external object has shifted to one or another of thesides of the infrared sensing assembly.

For example, if an external object is directly in front of the sensingassembly 104 as shown in FIG. 3, then the intensity of light received bythe photoreceiver 360 should be approximately the same regardless ofwhich of the phototransmitters (e.g., which of the phototransmitters352, 354, 356) is actuated (although at that close range, reflectedsignals are strong and tend to saturate the receiver). Correspondingly,if the signals received from the photoreceiver 360 are the same ornearly the same during each of three successive time windows duringwhich the three phototransmitters are successively actuated, thenprocessing of this information should determine that the external objectis in front of the sensing assembly 104. In contrast, if the receivedlight signal provided by the photoreceiver 360 during the time windowcorresponding to the actuation of the phototransmitter 352 is muchhigher than the received light signal provided by the photoreceiverduring the time window corresponding to the actuation of thephototransmitters 354 and 356, then processing of this informationshould determine that the external object is to the side of the sensingassembly 104, closer to the phototransmitter 352 than to either of theother two phototransmitters.

Although the above description of how to determine the location of anexternal object by way of triangulation particularly envisions the useof information concerning light received at a single photoreceiveroriginating at multiple phototransmitters (e.g., as is the case in theembodiments of infrared sensing assemblies shown in FIGS. 3 and 4), asimilar process is equally applicable where multiple photoreceivers areused to receive multiple different components of reflected light thatoriginated at a single phototransmitter (e.g., as is the case in theembodiment shown in FIG. 5). In all of these embodiments, to the extentthat multiple reflected light samples are obtained during a successionof time windows, it is typically assumed that the time windows aresufficiently short that it is unlikely that the external object willhave moved significantly during the overall span of time encompassingall of the time windows of interest. Also, while it can be the case thatsampling during a single set of time windows (e.g., where only one setof photoemissions has occurred, with each phototransmitter beingactuated only once) is adequate to determine the location of an externalobject, it is also possible that multiple repetitive reflected lightsamples will be obtained and utilized to determine the location of anexternal object (e.g., where the processing device not only takes intoaccount multiple samplings of received light occurring as each of thephototransmitters is successively actuated during successive timewindows, but also takes into account further samplings of received lightas the phototransmitters are successively actuated additional times).

Finally, notwithstanding the general description above of how reflectedlight information is utilized to determine an external object'slocation, it will be understood that other additional or differentprocessing steps can also be employed to determine or more closelyestimate object location. For example, in some circumstances, it isdesirable for background light determinations to be made prior to themaking of measurements of reflected light intensity (e.g., before or inbetween the successive time windows as discussed above), so thatbackground noise can be evaluated and taken into account by theprocessing device in its calculations, and/or so that the processingdevice can adjust operational parameters of the phototransmitters and/orphotoreceivers such as gain, etc. In this regard, for example, one canconsider the disclosures found in U.S. patent application Ser. No.12/344,760 filed Dec. 29, 2008 and entitled “Portable Electronic DeviceHaving Self-Calibrating Proximity Sensors” and U.S. patent applicationSer. No. 12/347,146 filed Dec. 31, 2008 and entitled “PortableElectronic Device Having Directional Proximity Sensors Based on DeviceOrientation”, each of which is hereby incorporated by reference herein,and each of which is assigned to the same beneficial assignee as thepresent application.

It should be further noted that, in at least some embodiments, operationof the sensing assembly can be limited so as to consider reflected lightonly originating from certain subset(s) of the availablephototransmitters. In some such embodiments where the sensing assemblyis implemented in a cellular telephone or PDA, a hand tracking/gesturingoffset to a side above the electronic device is enabled by eliminatingfrom the infrared tracking any signals originating fromphototransmitters on the side of the sensing assembly that is blocked asa result of the position offset. For example, with respect to theembodiment of FIG. 4, reflected light originating from one of thephototransmitters on a blocked side of the sensing assembly would not beconsidered in determining the presence/location of an external object(or possibly that phototransmitter would not be actuated to emit light).This manner of operation is workable because, if a human user places ahand above a touch screen and offset to the right so that the hand doesnot block a viewing of the touch screen, reflection from the left sideLED of the sensing assembly is almost nonexistent (point away andopposite to hand location) and the other three LEDs are used for handtracking and vice-versa (as a result, it is possible to track a hand bypositioning a hand to the side).

Turning to FIGS. 8 and 9, the positioning of a sensing assembly such asthe sensing assemblies 104, 400, and 500 of FIGS. 3-6 can vary dependingupon the embodiment and/or the electronic device. As shown in FIG. 8,for example, a sensing assembly such as the sensing assembly 400 can bepositioned at a location in the middle of the front surface of anelectronic device such as an electronic device 800. In some suchembodiments, the sensing assembly 400 can replace the navigation keycluster, such that the pyramid-type housing structure of the sensingassembly serves not only to house thephototransmitter(s)/photoreceiver(s) but also serves as abutton/actuator that can be pressed and/or tilted/rotated relative tothe front surface of the electronic device, thereby allowing forhands-free and/or touch-based control.

Also, notwithstanding the embodiment of FIGS. 1 and 6, a sensingassembly can be implemented at either end or along any edge of any givenelectronic device depending upon the embodiment. For example, as shownin FIG. 9, a sensing assembly 104, 400, 500 such as that of the FIGS.3-5 can be implemented at the opposite end of an electronic device(e.g., near the bottom of the front surface) 900 rather than at the endshown in FIGS. 1 and 6 (e.g., near the to of the front surface). Theelectronic device 900 also is intended to illustrate how a sensingassembly such as any of those described above can be implemented on anelectronic device in which the entire front surface is a glass orplastic/transparent video screen or touch screen. It should be notedthat blocking problems of the type discussed above (e.g., involving handpositioning) typically do not occur when the sensing assembly is at thebottom of a touch screen as shown in FIG. 9, albeit in such embodimentsit can be desirable to tilt the sensing assembly slightly toward a pointnearer to the center of the phone (or to use a lens to achieve sucheffect).

Although the above-described embodiments all envision the implementationof one or more photoreceivers and phototransmitters along (or recessedwithin) different walls of a pyramid-type structure, where therespective orientations of those photoreceiver(s)/phototransmitter(s)correspond to the orientations of the respective surfaces of thepyramid-type structure in which those devices are implemented, thepresent invention should also be understood as encompassing numerousadditional embodiments differing from those described above in certainaspects. For example, in at least some embodiments, thephotoreceiver(s)/phototransmitter(s), while being held together in amanner by which the various devices maintain relative angular positionsthat are the same as (or similar to) those described above, neverthelessare not housed within any particular pyramid-type housing structure withspecific walls as described above. Indeed, the present invention isintended to encompass embodiments in which there are merely severalphotoreceiver(s)/phototransmitter(s) that are assembled to one anotherbut have no walls or structures positioned in between those devices.

Also, the above-described embodiments envision particularly theimplementation of multiple (e.g., three or more) devices of one type(e.g., phototransmitters or photoreceivers) surrounding a single deviceof another type (e.g., a photoreceiver or phototransmitter), where thedevices of the one type are equally-spaced apart from one another aroundthe device of the other type, where the devices of the one type are allequally spaced apart from the device of the other type, and where thedevices of the one type are angularly offset in their orientationrelative to the orientation of the device of the other type by aconsistent angular amount (e.g., by the angle α or β), other embodimentsare also possible. For example, in some alternate embodiments, thedevices of the one type need not all be equally spaced apart from oneanother about the device of the other type, need not all be equidistantfrom the device of the other type, and/or need not all be offset intheir orientation relative to that of the other device by the sameamount.

In this regard, one exemplary alternate embodiment of a sensing assembly1000 is shown in FIG. 10. As shown, in this embodiment, the sensingassembly 1000 like the sensing assembly 400 of FIG. 4 has fourphototransmitters 1002 spaced around a single photoreceiver 354.However, in contrast to the sensing assembly 400, the phototransmitters1002 each are vertically oriented so as to have center axes oftransmission that are parallel to the center axis of reception of thephotoreceiver 354. That is, the phototransmitters 1002 are not at alloffset in their rotational orientation relative to the photoreceiver.Further, a housing 1006 within which the phototransmitters 1002 andphotoreceiver 1004 are supported does not necessarily have a pyramidalshape with any inclined surfaces.

Notwithstanding these differences between the sensing assembly 1000 andthe sensing assembly 400, the sensing assembly 1000 nonetheless is ableto transmit light and receive reflected light (as reflected by anexternal object) as if the phototransmitters were rotationally offsetrelative to the photoreceiver insofar as the sensing assembly 1000additionally includes a pyramid-shaped lens or prism 1008 (or possiblymultiple lenses in a pyramid-type shape) provided atop thephototransmitters and photoreceiver (or possibly only over one or moreof those devices) that refracts/bends the transmitted light exiting thesensing assembly/lens and/or refracts/bends the received light incidentupon the sensing assembly/lens, such that the overall transmission andreception of light out of and into the sensing assembly proceeds insubstantially the same manner as is experienced by the sensing assembly400. In some circumstances, the lens 1008 can be microfilm for beambending, particularly if the involved angles are small (e.g., 10 to 5degrees) and the photo-LEDs have relatively narrow transmission ranges(e.g., plus or minus 30 degrees). Although the lens 1008 is shown to beof a pyramid-type form that includes four inclined sides sloping awayfrom a tip of the lens (in this case, this tip can be considered acentral surface of the lens), in other embodiments, the lens can take aform that is more similar to that of the pyramid-type structuresdescribed above in relation to FIGS. 3-5, in which the tip portion ofthe pyramid is missing such that there exists a central surface that ismore extensive (e.g., such as the top surfaces 348, 482 and 510) awayfrom which the inclined surfaces slope.

The present invention further is intended to encompass additionalembodiments of sensing assemblies that are particularly useful forimplementation in certain types of electronic devices. Referringparticularly to FIG. 11, a further sensing assembly 1100 is shown to beimplemented in relation to a glass (or transparent plastic) video screenor touch screen 1102 as is common in certain types of electronicdevices, including for example the electronic device 900 of FIG. 9. Asshown, in the embodiment of FIG. 11, the sensing assembly 1100 includesfour transceivers 1104, each of which includes a respectivephototransmitter and a respective photoreceiver, and the respectivetransceivers are respectively positioned at the midpoints of each of thefour side edges of the screen 1102, respectively. Further as shown, thesensing assembly 1100 also includes a pyramid-type formation 1114 thatis formed as part of (or positioned just beneath) the screen 1102. Thepyramid-type formation 1114 includes four inclined surfaces 1108extending from the four sides of a square top (horizontal) surface 1106,where each of the inclined surfaces slopes downwardly from the topsurface towards one of the respective edges of the screen 1102.

The sensing assembly 1100 of FIG. 11 operates as follows. In a firstmanner of operation, light is transmitted from each of thephototransmitters of the respective transceivers 1104 via respectiveoptical waveguides 1110 through the screen 1102 (or just beneath thescreen, parallel to its surface) toward the respective one of theinclined surfaces 1108 closest to that respective transceiver. Uponreaching the inclined surfaces, the light is reflected outward from thesensing assembly 1100 (and thus from the electronic device on which itis implemented) at various angles depending upon the slopes of theinclined surfaces 1108, with the light transmission being centered aboutrespective center axes of transmission 1112. Thus, transmitted lightemanates from the sensing assembly 1100 in much the same manner as ifthe light had been emitted directly from phototransmitters arrangedalong the sides of a pyramid-type structure as shown in FIG. 4. Afterthe light is emitted about the center axes of transmission 1112, it canthen be reflected off of an external object such as the hand 111 ofFIG. 1. Portions of the reflected light eventually are received by oneor more of the photoreceivers associated with the respectivetransceivers 1104, and thereby the reflected light is sensed.

Further variations of the sensing assembly 1100 are also possible. Forexample, in one alternate embodiment, rather than reflecting light to betransmitted out of the sensing assembly, the inclined surfaces 1108 ofthe pyramid-type formation 1114 instead are intended to reflect incomingreflected light back toward the transceivers 1104, at which are locatedrespective photoreceivers. In such embodiments, the phototransmitters ofthe transceivers 1104 can be configured to transmit light directlyoutward (e.g., perpendicular to the surface of the screen 1102) at thelocations of the transceivers, with that light in turn being partly orentirely reflected by an external object back toward the pyramid-typeformation 1114. In further alternate embodiments, rather than employingfour transceivers that each have a respective phototransmitter and arespective photoreceiver, only four phototransmitters or fourphotoreceivers are provided at the locations of the transceivers 1104shown in FIG. 11. In such embodiments, where four phototransmitters arepositioned at the edges of the screen 1102, a photoreceiver can bepositioned along the top surface of the pyramid-type formation and,where four photoreceivers are positioned at the edges of the screen, aphototransmitter can be positioned along the top surface of thepyramid-type formation.

Each of the embodiments described above in relation to FIG. 11 areparticularly advantageous insofar as they allow for the use of apyramid-type formation such as the pyramid-type formation 1114 having aheight that is considerably less than the heights of the pyramid-typeformations of the sensing assemblies 104, 400, 500 described above.Thus, there is no need (or much less need) to have a housing structureprotruding outward from the surface of the electronic device. Furtherthe pyramid-type formation 1114 can be transparent and thussubstantially the same in appearance as the remainder of the screen1102. Thus, the use of such pyramid-type formations such as theformation 1114 can be particularly advantageous for use in electronicdevices where it is desired that the front surface of the device be alarge flat video screen or touch screen, uninterrupted by bumps orregions where the video screen or touch screen is unable to displayinformation.

It should be noted with respect to the sensing assembly embodiments ofFIGS. 10-11 that, even though the structures employed are different tosome extent than those shown in FIGS. 1-6, each of these embodimentsnevertheless can be operated in essentially the same manner as isdescribed with reference to FIG. 7. Further, although the lens 1008 ofFIG. 10 and the pyramid-type formation 1114 of FIG. 11 are four-sidedpyramid-type structures, in other embodiments other pyramid-typestructures (e.g., tetradhedral structures) can also be employed. In somecases, a pyramid structure is not necessary, because thephototransmitters and/or photoreceivers can be appropriately tilted suchthat light is emitted in desired directions.

Notwithstanding the above discussion, the present invention is intendedto encompass numerous other embodiments as well. For example, in someother embodiments, there are only two phototransmitters (and one or morephotoreceivers) or only two photoreceivers (and one or morephototransmitters). In other embodiments, there are more than fourphototransmitters (and one or more photoreceivers), or more than fourphotoreceivers (and one or more phototransmitters). Also, while in manyembodiments of the present invention the sensing assembly is intended tobe mounted to an electronic device in a fixed/stationary manner, whichcan be advantageous because such manner of mounting can be easilyachieved without the need for many complicated components, in some otherembodiments it is possible that the sensing assembly is mounted to anelectronic device in a tiltable, rotational, or translatable manner toallow for tilting, rotation and/or translation of the sensing assemblyrelative to the remainder of the electronic device (typically, suchtilting, rotation and/or translation would be limited in nature, e.g.,as discussed above in the example where the sensing assembly replacesthe navigation key cluster). Additionally, while in some embodimentsdiscussed above such as those of FIGS. 3 and 4 the photoreceiver(photodiode) is placed inside the pyramid-type structure (e.g., at thecenter of the structure), in alternate embodiments the photoreceiver(photodiode) can be positioned on top of or outside of the pyramid-typestructure or its center.

Further, although the embodiments discussed above envision a singleinfrared sensing assembly being implemented on a given electronicdevice, it is also possible in some other embodiments that multipleinfrared sensing assemblies will be implemented on a given electronicdevice. For example, in some embodiments of electronic devices, twosensing assemblies positioned on diametrically-opposed outer surfaces ofthe electronic device can be employed so as to allow for the detectionof the presence and location of external objects on both sides of theelectronic device. Additionally, although the particular tetrahedron andfour-sided pyramid structures are described above, it should beunderstood that other embodiments employing similar structures havingmultiple inclined surfaces and the like are also encompassed within thepresent invention. Further, while the use of a lens/pyramid structurefor the purpose of bending/refracting light is discussed above withrespect to certain embodiments, the bending/refracting of light can alsobe achieved by having an optical diode placed in a tilted package, orhaving a tilted lens attached to it (indeed, in some circumstances aninfrared photo-LED or photodiode for use as a phototransmitter orphotoreceiver will be manufactured by a vendor with such tiltedcharacteristics, which can for example be referred to as “top shoot”,“side shoot”, or “tilted shoot”, among other things).

Also, while in the embodiments discussed above it is envisioned that thesensing assembly will be implemented in conjunction with an electronicdevice or other device, where the electronic device or other device willinclude the processor and/or other components appropriate forcontrolling actuation of the phototransmitter(s) of the sensingassembly, for receiving signals indicative of the receiving of reflectedlight by the photoreceiver(s), and for determining the presence andlocation of external object(s) based upon those received signals, inother embodiments it is possible that the sensing assembly will itselfinclude processor and/or other components as are appropriate (e.g.,memory device(s), battery/power source device(s), and input/outputterminal(s), etc.) for allowing the sensing assembly to operate byitself in terms of controlling the actuation of its phototransmitter(s),monitoring the operation of its photoreceiver(s), makingpresence/location determinations, and communicating suchpresence/location information to other external devices. In some suchembodiments, the sensing assembly itself has one or moreterminals/ports/interfaces suitable for allowing the sensing assembly tocommunicate with remote devices via wired or wireless networks includingby way of internet-type networks.

Embodiments of the present invention allow for an electronic device,with an appropriate sensing assembly, to achieve beneficial manners ofoperation based upon the information obtained regarding the presence andlocation of external object(s). For example, in some electronic devicessuch as cellular telephones, the presence and location of a human user'sphone is of interest and can be used to govern or influence one or moreoperations of the phone. To begin, the use of a sensing assembly such asthose described above can allow a mobile phone to detect whether a humanuser's hand or ear are proximate a right side of the phone or a leftside of the phone, and thereby allow for appropriate adjustments tophone operation. Further for example, the volume of a phone speaker canbe automatically adjusted based upon the sensed position of a humanuser's head. Sensing assemblies such as those described above also canenable tracking movement without blockage when placing/tracking a handabove the phone offset to the left or right side of the phone.

Also for example, through the use of a sensing assembly such as one ormore of those discussed above, it is possible to enable an electronicdevice to sense and recognize hand gestures that signify user selectionsor commands. Further for example in this regard, sensed movement of afinger of a human user above the front surface of an electronic devicecan signify a command by the human user that an image or contentdisplayed on the electronic device be paused/frozen (e.g., to facilitatesending or sharing of the image/content), changed, free/selected (e.g.,that a page of information be turned so that a different page ofinformation is displayed), shared, etc., or that a cursor displayed on ascreen be moved (e.g., a command such as that often provided by a“mouse”), or that a zoom level or pan setting regarding an image (e.g.,a map or photograph) be modified. In this manner, such infraredgesturing can serve as a substitute for a touch screen, where a userneed not actually touch the surface of the electronic device to executea command (albeit the system can still be implemented in a manner thatalso allows for commands to be recognized when touching does occur). Byeliminating the need to touch a screen, disadvantages potentiallyassociated with touching (e.g., fingerprints and other smudging of avideo display screen or germ transmission) can be reduced.

In some circumstances, different hand movements or repeated handmovements sensed by way of the sensing assembly of an electronic devicecan be understood as constituting a first command that a particularvariable operational characteristic be selected (e.g., that a volumecontrol icon appear on the video screen of the electronic device)followed by a second command modifying a setting of the variableoperational characteristic (e.g., that the volume be set to a particularlevel). Particularly in this regard, for example, because infraredsensing assemblies of the type described above are capable of detectingboth movements across the assemblies (e.g., horizontal xy-planemovements) as well as movements toward or away from the assemblies(e.g., vertical z-axis movements), a horizontal-plane gesture can befollowed by a vertical axis gesture as an indication of particularcommands. Further for example, using such gestures, the horizontalgesture could precipitate a volume (or zoom) adjustor icon to becomeavailable while the vertical gesture could in fact cause adjustment inthe volume (or zoom) to a desired level. Alternatively, where multiplerepeated hand movements are anticipated, the failure of a second orsuccessive hand movement to occur can be interpreted as a command thatsome other action be taken (e.g., that a cursor or image be recenteredor otherwise repositioned).

One example of operation encompassing a number of the above-describedconsiderations would be as follows. Suppose a user placed a handapproximately six inches above a touch screen and to the right side of acellular telephone on which an infrared sensing assembly was provided.Immediately, in this instance, the phone might respond by placing acursor on the right side edge of the touch screen corresponding to thehand location. However, assuming that the user hand was kept stationaryin that location for one second, then the phone might further act tore-center/map the cursors to the middle of the touch screen(corresponding to the hand being near the right side of the phone). Asdiscussed above, given placement of the hand on the right side of thephone, the phone might operate to track the hand by operating thesensing assembly so that only certain portions of reflected light (e.g.,as generated by certain ones of the phototransmitters, for example,three out of four of the phototransmitters of the sensing assembly ofFIG. 4, but not the phototransmitter pointing toward the left side ofthe phone) were considered. Once the user completed an operation ofinterest (e.g., panning or zooming), the user's hand might remainstationary again and this could signify that the current image should bepaused/frozen.

In some embodiments the operation of existing other sensors of anelectronic device (e.g., an accelerometer capable of detecting aphysical tapping of a navigation key cluster) can be coordinated withthe operation of an infrared sensing assembly such as those describedabove. Indeed, depending upon the embodiment, a variety of other sensorsin addition to an infrared sensing assembly can be utilized in detectingcommands in a navigation mode of operation and/or to adjust an infraredrange accordingly in switching between an infrared sensing mode ofoperation and a touch-based mode of operation. For example, in someembodiments in which the sensing assembly is implemented as a navigationkey cluster, navigation can be achieved by a hand gesture above thesensing assembly (not touching the sensing assembly), followed bypressing of the center of the navigation device to achieve selection. Insuch a case, infrared reception would go from a maximum level (where thefinger was near the sensing assembly) to a minimum level (where thefinger blocks reception entirely), and such a maximum to minimumoccurrence would be interpreted as constituting a selection input.Alternatively for example, a tap as sensed by another sensor could thenprecipitate the electronic device's anticipating an imminent usercommand that would be sensed via the infrared sensing assembly. Also, insome circumstances, sliding of an external object such as a fingerdirectly along the sensing assembly (involving touching) can berecognized as a command.

Electronic devices implementing sensing assemblies such as thosedescribed above can be utilized in other contexts as well. For example,an electronic device implementing a sensing assembly can be operated soas to recognize the proximity of a surface (e.g., a desktop) to theelectronic device, such that the electronic device when positioned andmoved over the surface can be utilized as a mouse. Relatedly, by sensingthe positioning/tilting of a human user's hand relative to an infraredsensing assembly on an electronic device, mouse-type commands can alsobe provided to the electronic device. In such applications, it can beparticularly desirable to utilize phototransmitters having narrowangular ranges of transmission to allow for high sensitivity indetecting the tilting of a user's hand.

Also, in some embodiments, operation of the sensing assembly itself canbe controlled based upon sensed information concerning the location ofexternal object(s). For example, in some cases, the sampling rate (e.g.,in terms of the frequency with which the various phototransmitters of asensing assembly such as the sensing assembly 104 are actuated to emitlight) can be modified based upon the proximity of the user, so as toadjust the sensitivity of the location detection based upon theproximity of the user. Indeed, while the manner of operation describedwith respect to FIG. 7 envisions that the different phototransmitters ofa given sensing assembly will be actuated in succession rather thansimultaneously, in some cases it may be desirable to actuate all of thephototransmitters simultaneously to increase the overall intensity ofthe light emitted by the sensing assembly, which can increase theoverall amount of reflected light that makes its way back to thephotoreceiver and thereby make it possible to sense the proximity of anexternal object even though the object is a fairly large distance awayfrom the sensing assembly. For example, the range of proximity detectionof a sensing assembly can be increased from six inches where thephototransmitters are successively actuated to two feet where all of thephototransmitters are actuated simultaneously (this can be referred toas “super-range proximity detection”).

More specifically with respect to the detection of gestures, a sensingassembly such as sensing assembly 104, 400, or 500, in conjunction witha processor, such as processor 204, can be used to detect one or more ofvarious basic gestures, where each gesture is a predefined movement ofan external object (such as a user's hand or thumb or finger) withrespect to the electronic device, and to control operation of theelectronic device based upon the detected gesture. Operation of theelectronic device can also be based upon a determination of a locationof the object at various times during the gesture. The sensing assemblyand processor can detect the presence and movement of objects in a threedimensional space around the sensing assembly, and so the variousdifferent gestures can be defined as movements in this three dimensionalspace rather than in a one or two dimensional space.

The various predefined basic gestures to be detected can include forexample, a push/pull gesture (negative or positive z-axis movement), aslide gesture (xy planar movement), a hover gesture (stationaryplacement), and a tilt gesture (rotation of the external object about acorresponding pitch, roll, or yaw axis), as well as differentcombinations of these four basic gestures. The sensing assembly andprocessor can be operable to run a specific routine to detect acorresponding one of these gestures, and/or to detect and distinguishbetween two or more predefined gestures. Each predefined gesture(including a combination gesture) can be associated with a respectivepredetermined control operation of the electronic device. In some cases,determined locations of the object at corresponding times of a gesturecan be used such as to control a particular setting of a controloperation.

As mentioned above, the gestures can be defined to be performed in atouchless manner (i.e., without touching a display screen or the like ofthe electronic device), although some can involve touching of theelectronic device. Further, the gestures can be defined to have apredetermined start or end location, or other orientation with respectto the electronic device or sensing assembly. For example, certaingestures can be defined to be performed in an “offset” manner withrespect to a display screen, in order for the display screen to remainunobstructed by movement of the object.

With respect to examples of predefined gestures, FIGS. 12-14sequentially illustrate a push gesture performed by movement of anobject, in this case a user's hand 111, toward an electronic device 1200(such as a mobile device) having a sensing assembly such as sensingassembly 400. More specifically, using the three dimensional (3D)coordinate system illustrated, a push gesture can be defined to bemovement of an object in a negative z direction from a first position asshown in FIG. 12, to a second position closer to the sensing assembly400, such as shown in FIG. 14. In this case, the user's hand is shown asbeing generally centered above the sensing assembly 400, although thisis not necessary for the detection of a push gesture. Similarly, a pullgesture can be defined to be movement of an object in a positive zdirection from a first position close to the sensing assembly to asecond position farther away from the sensing assembly. As describedbelow, a z distance calculation routine can be utilized to determine theapproximate distance between the object and the electronic device duringone or more time periods of the push or pull gesture.

Generally a slide or swipe gesture can be defined to be movement of anobject in a defined plane across the electronic device, and preferablyat a generally constant distance from (typically above) the electronicdevice. For example, FIGS. 15-17 sequentially illustrate a side-to-sideslide gesture performed by movement of a user's hand 111 in the xy planeand in a negative x direction (as indicated by arrow 1502) from a firstside 1504 of electronic device 1200, across the electronic device andpreferably across the sensing assembly 400, to a second side 1506 of theelectronic device 1200. Similarly, a top-to-bottom (or bottom to top)slide gesture can be defined by movement of an object across the sensingdevice such as from a top side of the electronic device in a negative ydirection to a bottom side of the electronic device, or in a positive ydirection from bottom to top. Various other slide gestures can also bedefined which occur in a specified direction in the defined xy plane. Apartial slide gesture can be defined to be movement that extends onlypartially across the electronic device. A general xy location of theobject with respect to the electronic device can be determined atdifferent time periods of the slide gesture.

A hover gesture can be defined to be no movement of an object, such as adownward facing hand, for a certain period of time, such as one or moreseconds. A cover gesture can be defined to be a special case of a hovergesture, such as where an object such as a cupped hand is touching theelectronic device and substantially covers the sensing assembly. A tiltgesture can be defined to be rotation of an object such as a hand abouta roll axis (x axis), a yaw axis (y axis), or a pitch axis (z axis).

Combination gestures, such as a dive or swoop gesture, can be defined tobe a push gesture immediately followed by a tilt gesture. For example, adive gesture can be defined by an object such as a hand which movescloser to the sensing assembly with fingers initially extended generallytowards the electronic device (push gesture in −z direction) and whichthen changes to fingers extended generally parallel to the electronicdevice (in the xy-plane via a tilt gesture such as around an axisparallel to the x axis).

Certain gestures can be defined to be performed by a hand in a specifichand or finger configuration and the sensing assembly and processor canfurther operate to detect in certain circumstances a specific handconfiguration in conjunction with a predefined gesture. For example, onesuch gesture can be a slide gesture performed by a hand palm side facethe sensing assembly and with two extended fingers (such as in a peacesign configuration). Various other gestures and hand configurations canalso be defined.

Basically in order to detect gestures, one or more phototransmitters ofthe sensing assembly are controlled by the processor to emit light oversequential time periods as a gesture is being performed, and one or morephotoreceivers of the sensing assembly receive any light that is emittedfrom a corresponding phototransmitter and is then reflected by theobject (prior to being received by a photoreceiver) to generate measuredsignals. The processor, which preferably includes an analog to digitalconverter, receives these measured signals from the one or morephotoreceivers, and converts them to a digital form, such as 10 bitdigital measured signals. The processor then analyzes all or a portionof these digital measured signals over time to detect the predefinedgesture, and to perhaps determine a specific hand configuration, and toperhaps determine one or more relative locations of the object duringone or more corresponding times of the gesture. The analysis can beaccomplished by determining specific patterns or features in one or moreof measured signal sets or modified or calculated signal sets. In somecases, the timing of detected patterns or features in a measured signalset can be compared to the timing of detected patterns or features inother measured signal sets. In some cases, distances along the z axis,xy locations, and/or the amplitudes of detected patterns or features canbe determined. Other data manipulation can also be performed. Thepredefined basic gestures can be individually detected or can bedetected in predefined combinations, allowing for intuitive and complexcontrol of the electronic device.

FIG. 18 is an exemplary method for detecting a predefined basic gestureand can be used with a sensing assembly like any of those describedabove, including one having multiple phototransmitters and at least onephotoreceiver, or one having multiple photoreceivers and at least onephototransmitter, or one having multiple transceivers (with or without apyramid structure). In the case of multiple phototransmitters which cansurround a single photoreceiver, as described above, each of thephototransmitters is oriented such that it emits infrared light outwardaway from the electronic device about a corresponding centraltransmission axis, with each central transmission axis extending in adifferent direction with respect to the sensing assembly and electronicdevice. In this manner, a large portion of the volume adjacent to theelectronic device can be reached by emitted infrared light in order toallow the movement of an object to be tracked across this volume. Asimilar ability to track movement of an object exists with a sensingassembly having multiple photoreceivers which can surround a singlephototransmitter or with a sensing assembly having multiple transceivers(wherein each transceiver essentially includes a phototransmitterco-located with a photoreceiver).

In particular, the exemplary method begins at step 1800, which is aninitiation for indicating that a gesture detection routine should bestarted. Initiation can be accomplished in a number of ways such as whena user launches or focuses on a particular application on the electronicdevice, a particular portion or step of an application, or when a userindicates gesture detection should be initiated using one of the variousinput devices of the electronic device in a predetermined manner, or bya combination of these steps. The processor can be capable of performingvarious gesture detection routines individually or simultaneously.

At a step 1802, the processor controls the phototransmitter(s) tocontrol the timing and intensity of the infrared light emitted by thephototransmitter(s). For example, if the sensing assembly includes asingle phototransmitter, the phototransmitter is controlled to emitlight during each of multiple sequential time periods as the externalobject moves in the specified pattern of movement. If the sensingassembly includes multiple phototransmitters, each of thephototransmitters can be controlled to emit light during a respective,non-overlapping, portion of each of multiple sequential time periods asthe external object moves in the specified pattern of movement. In thismanner, each measured signal generated by a photoreceiver can beassociated with a respective one of the phototransmitters. The length ofa time period is preferably selected such that the amount that an objectmoves during the time period is negligible as compared to the totalmovement of the object for a complete gesture. In some cases asdescribed above, the phototransmitters can each emit light at differentfrequencies (wavelengths), or bandwidths, and these phototransmitterscan then be controlled to transmit light at the same time during each ofthe time periods. The benefit of the phototransmitters transmitting atthe same time is enhanced speed.

At a step 1804, measured signals indicative of intensity of receivedlight are generated by the photoreceiver(s). For example, assuming thatthe sensing assembly includes multiple phototransmitters and at leastone photoreceiver, then for each phototransmitter and for each timeperiod, a corresponding measured signal can be generated by thephotoreceiver which is indicative of a respective amount of infraredlight which originated from that corresponding phototransmitter duringthat corresponding time period and was reflected by the external objectprior to being received by the photoreceiver. If the phototransmitterstransmit light at the same time, then the measured signals can bedecoded such as by frequency filtering or the like, in order to discernwhich signals originated from each of the different phototransmitters.This can also be accomplished with the use of multiple photoreceivers.

In another example, wherein the sensing assembly includes multiplephotoreceivers and at least one phototransmitter, for each of theplurality of photoreceivers and for each of the plurality of sequentialtime periods, a corresponding measured signal can be generated which isindicative of a respective amount of infrared light which originatedfrom the phototransmitter during the corresponding time period and wasreflected by the external object prior to being received by thecorresponding photoreceiver.

As described below, the intensity of the emitted infrared light can becontrolled to ensure that the photoreceivers are not saturated so thatthe measured signals provide useful information.

The measured signals are preferably digitized by an A/D converter toprovide sets of digital measured signals, with each digital measuredsignal set corresponding to a respective phototransmitter (such as inthe case of multiple phototransmitters and a single photoreceiver) or arespective photoreceiver (such as in the case of multiple photoreceiversand a single phototransmitter). The digital signals can also becorrected to take into account non-zero values obtained when acorresponding phototransmitter is not emitting light. This entails theacquisition of one or more measured signals when no phototransmitter istransmitting and the subtraction of this value from the digital valuesto produce compensated digital signal values. For example, assuming useof a sensing assembly such as sensing assembly 400 shown in FIG. 4,which includes a single photoreceiver 492 surrounded by fourphototransmitters 484, 486, 488, and 490, a background reading from thephotoreceiver 492 can be initially obtained when no phototransmitter istransmitting, and then each phototransmitter can be pulsed on one at atime and four corresponding measured intensity signals or readings areobtained corresponding to one time period. These four readings can becompensated by subtracting the background reading and this procedure canbe repeated for each subsequent time period.

In order to provide meaningful measurements through an entire range ofpossible object locations, an automatic power control scheme can beimplemented to control the intensity of emitted infrared light in step1802 to avoid saturation of the photoreceiver(s). The followingdescription again assumes use of sensing assembly 400 as shown in FIG.4, i.e., with multiple transmitters and a single photoreceiver, however,analogous operation applies to other sensing assembly embodiments.Basically, the power control scheme operates by obtaining correspondingmeasured signals with the phototransmitters operating at one of variouspower settings during at least one time period and checking that thephotoreceiver is not producing signals at the top of an output rangeduring this time period. For example, three different power settings canbe employed for the phototransmitters: a high setting, a medium setting,and a low setting. Respective measured signals from the photoreceivercorresponding to each of the phototransmitters are first obtained withthe phototransmitters controlled to emit light at the high settingduring a time period (where the phototransmitters can be controlled toemit light at respective portions of the time period if they emit lightat the same frequency or bandwidth, and where the phototransmitter canbe controlled to emit light at the same time during the time period ifthey emit light at different frequencies or at different bandwidth). Ifthe measured signals indicate no saturation, these signals are used insubsequent calculations corresponding to that time period. If themeasured signals corresponding to the high setting are saturated, thenadditional measurements in a subsequent time period are taken at themedium power setting. If the measured signals corresponding to themedium setting indicate no saturation, then these signals are used insubsequent calculations. If the measured signals corresponding to themedium setting indicate that the photoreceiver is saturated, thenadditional measurements are taken at the low power setting in asubsequent time period and these are used in subsequent calculations.The low power setting is set up to produce measured signals just belowsaturation when the photoreceiver is completely covered by an object atthe surface of the sensing assembly. This procedure can be repeated foreach of the time periods needed to detect a gesture.

As noted, the measured digital signals are a measure of the intensity ofthe reflected infrared light. The power levels can be chosen to providesome overlap between levels such that the measured signals fromdifferent power levels can be converted to a standard scale such thatthey can be combined together into a continuous curve. For example, datacan be taken for the overlap regions (such as corresponding to severalpush or pull gestures) and a curve fit performed. In one example, thefollowing equations are obtained for converting measurements obtained atthe various power levels to a standard intensity scale denoted by I:I=I_(PowerLevel=high)I=12*I _(PowerLevel=medium)+38I=128*I _(PowerLevel=low)+3911

In the above manner, measured signal sets can be obtained which provideintensity values over time corresponding to the differentphototransmitters emitting light in different directions orcorresponding to the different photoreceivers receiving light fromdifferent directions. Each digital measured signal set can providerelevant information regarding the presence or absence of an object in arespective volume corresponding to a respective phototransmitter orphotoreceiver and relative to the sensing assembly.

At a step 1806, one or more of the measured signal sets are evaluated todetect the predefined gesture and to determined corresponding locationsof the object at various times during the gesture. For example, asfurther described below, a specific feature of a measured signal set canbe sought and the timing of this feature can be compared with the timingof a corresponding feature in one or more of the other measured signalsets to detect the gesture. Furthermore, as also described below, one ormore of the measured signal sets, or portions thereof, can be combinedin a specified manner and evaluated so as to extract relevantinformation regarding the occurrence of a gesture.

At a step 1807, a determination is made as to whether the gesture hasbeen detected. If so, processing proceeds to a step 1808, and if not,processing proceeds to a step 1809. At step 1809, a request is generatedfor a user to repeat the gesture, and processing then proceeds to step1802.

At the step 1808, the operation of the electronic device is controlledin response to the detected gesture, such as by controlling a specificfunction of the electronic device or controlling the selection ofcontent stored on the electronic device. The various predefined gesturescan each be associated with any one of a variety of electronic deviceoperations, although preferably, the predefined gestures each control anoperation or action of the electronic device in an intuitive manner. Forexample, the detection of a push gesture can be used to decrease orlimit a function, such as to turn down the volume for a music player, orperform a zoom operation for a camera feature of the electronic device,wherein the distance of the object from the electronic device at aspecified time can be correlated to the amount that the volume or zoomoperation will be changed. Similarly, a pull gesture can be used tocorrespondingly increase a function. Push and pull gestures can also beused to navigate through stacked menus, pictures, or other items forselection.

As another example, a slide gesture over the display screen from top tobottom can denote an erasure or closing of an application, while a slidegesture from side to side of the display screen may indicate a scrollfunction, or the like, wherein a relative xy location of the objectduring the slide gesture is linked to the position of a cursor on adisplay screen of the electronic device. A hover gesture, especially inconjunction with other gestures for locating an item can mean aselection of an item after it has been located, such as the selection ofa specific file, image, song, or other item. A tilt gesture about a yaxis for example, can denote the page turning of an e-book or photoalbum.

Advantageously, a specific gesture (including a specific combinationgesture) can be used to easily and quickly select one or more itemsdisplayed on the display screen of the electronic device in a touchlessmanner. Because predefined gestures are detectable in a threedimensional space, this allows for various menus or displays of itemssuch as contacts or pictures to be arranged in a quasi three dimensionalmanner on a display screen of the electronic device. Specific itemsselectable through the use of one or more predefined gestures includingpush/pull, slide, tilt, and hover gestures for controlling the movementof a corresponding cursor or other selection device through the threedimensional arrangement of items. For example, if several groups of twoor more stacked windows (or photos or documents or other items) areshown on the display screen of the electronic device, a user can performone or more slide gestures to select a desired group, followed by a pushgesture to maneuver within the stack. Alternately, a user can perform aslide gesture to push one or more top windows out of the way, or a usercan reach a hand toward the screen with a push gesture followed by atilt gesture to dive past one or more top windows and slide a lowerwindow out to the side for better visibility.

As mentioned above, various gesture detection routines including variousprocessing steps can be performed to evaluate the measured signals. Forexample, assuming the use of a sensing assembly 400 as shown in FIG. 4,FIG. 19 shows an exemplary graph of intensities versus time curves 1900,1902, 1904, and 1906 which represent digital measured signal setscorresponding to respective phototransmitters 484, 486, 488, and 490 fora push gesture. Basically, as an object moves closer to the sensingassembly 400, the corresponding intensity values in each set increaseduring the same time frame (which includes a plurality of sequentialtime periods), and if the object is generally centered above the sensingassembly as the gesture is performed, the amount that each set of valuesis increased over that time frame is generally the same, as shown inFIG. 19.

In cases where the object is offset somewhat from the sensing assembly,minimum intensity values and maximum intensity values (correspondingrespectively to when the object is at a far distance and when the objectis at a near distance) of the measured signal sets would still occur atroughly the same respective times, but would have different values(amplitudes) at the same respective times as between the different sets.For example, FIG. 20 is an exemplary graph of intensities versus timecurves 2000, 2002, 2004, and 2006 which represent digital measuredsignal sets corresponding to the respective phototransmitters 484, 486,488, and 490 for a pull gesture, and illustrates that as an object movesfarther away from the sensing assembly, the corresponding intensityvalues of the measured signals sets all decrease during the same timeframe. If the object is generally centered above the sensing assembly asthe gesture is performed, the amount that each set of values isdecreased over the time frame is generally the same amount. However, asshown in FIG. 20 in a case where an object is offset somewhat from thesensing assembly such as by being generally centered to the right sideof the sensing assembly 400, then maximum and minimum intensity valuescorresponding to each of the measured signal sets would still occur atroughly the same respective times, but would have differing values. Inthis case, if the object is generally centered to the right side of thesensing assembly 400, then the measured signal set corresponding to thephototransmitter on the right side, namely phototransmitter 486, willhave the largest values, measured signal sets corresponding tophototransmitters 484 and 488 will generally track together, and themeasured signal set corresponding to phototransmitter 490, which isfarthest away from the object and directs light away from the object,will have the smaller values as compared to the other. Note thatintensity is related to distance in an inverse, non-linear manner, andassuming that a push or pull gesture is performed at an approximatelyconstant speed, the intensity values will increase or decrease in anon-linear manner.

Therefore, a gesture detection routine for detecting a push (or pull)gesture can include steps to evaluate one or more of the measured signalsets to determine whether corresponding intensity values are increasing(or decreasing) over time, and can include steps to compare amplitudesof these sets with respect to each other at one or more times. Thenumber of different measured signal sets to be evaluated can be based onwhether other gestures need to be detected and distinguished and whichother gestures these may be. For example, if just a push gesture is tobe detected, then evaluation of a single measured signal set can besufficient to determine if intensity values are sequentially increasing,while if it is desired to distinguish between a generally centered pushgesture and an offset push gesture, then two or more of the measuredsignal sets would need to be included in the analysis.

Processing steps can be performed on the digital measured signal sets toconvert intensity values to corresponding distances. In particular, theprocessor can be programmed to perform a Z distance calculation routineusing the measured digital signals to determine an object's relativedistance above the central surface (or other reference surface on theelectronic device) at one or more different times during a push or pullgesture. Because the intensity of the measured reflected light (i.e.,the measured signal) is dependent upon the size, color, and surfacetexture/reflectivity of the object, an exact value for distance can notbe determined based solely on the received intensity, but the electronicdevice can be calibrated so as to provide an approximate distance basedon the use of a specific object, such as an open medium-sized hand.Alternately, the user may perform a calibration routine to personalizefor the user's individual left or right hand.

Specifically, the reflected light intensity varies as a function of1/distance². A resulting distance or Z value corresponding to each ofthe phototransmitters can then be calculated and scaled to be within acertain range based on a measured intensity value. For example, assumingfour phototransmitters, distance values Z₁, Z₂, Z₃ and Z₄ correspondingto a respective phototransmitter can be calculated as a 10 bit valuewithin a predetermined range, such as a value between 0 and 1000 (withany results greater than 1000 being set to 1000) using the followingequation using a measured intensity I:Z=10000/sqrt(I)

In some cases, an average Z value representing distance can then becalculated by averaging together the Z values calculated correspondingto the multiple phototransmitters, such as:Z _(avg)=(Z ₁ +Z ₂ +Z ₃ +Z ₄)/4In some cases, distances can be calculated using corresponding measuredsignals from a subset of all the phototransmitters (or photoreceivers).

In one embodiment, the processor can be programmed to calculatecorresponding distances for each of the sequential time periods of apush or pull gesture. For a push gesture, these distances aresequentially decreasing over time (in a generally linear manner assuminga constant speed of the push gesture), and for a pull gesture, thesedistances are sequentially increasing over time. In this manner, it ispossible to associate a corresponding calculated distance with theposition of a cursor such as to locate a particular item in a stack ofitems on a display screen of the electronic device, or to associate acorresponding calculated distance with a particular change in or amountof change of a control setting, such as for a volume or zoom controlfunction.

With respect to a slide gesture, assuming that a z-axis distance of theobject from the sensing assembly remains relatively constant, then theoccurrence of a slide gesture and its direction can be determined byexamining the timing of the occurrence of intensity peaks incorresponding measured signal sets with respect to one or more of theother measured signal sets. As an object gets closer to a specificphototransmitter's central axis of transmission, the more light fromthat transmitter will be reflected and received by a photoreceiver, suchas the photoreceiver 492 of sensing assembly 400 shown in FIG. 4. Thetiming of the intensity peaks in each measured signal set with respectto the other measured signal sets provides information regarding thedirection of travel of the object. For example, FIG. 21 is an exemplarygraph of intensity versus time curves 2100, 2102, 2104, and 2106, whichrepresent measured signal sets corresponding to respectivephototransmitters 486, 484, 488, and 490 for a slide gesture performedby an object such as a hand that moves above sensing assembly 400 ofFIG. 4, and specifically illustrates a slide gesture of an object movingfrom the right side to the left side across the electronic device. Thus,the object is first closest to phototransmitter 486, then moves acrossphototransmitters 484 and 488 at roughly the same time, and is thenclosest to phototransmitter 490.

Similarly, FIG. 22 is an exemplary graph of intensities versus timecurves 2200, 2202, 2204, and 2206 for a slide gesture by an objectmoving from top to bottom across the sensing assembly 400 (denoted hereas a vertical gesture), wherein the curves 2200, 2202, 2204, and 2206represent measured signal sets corresponding to respectivephototransmitters 484, 486, 490, and 488. In this case, the object movestop to bottom across phototransmitter 484 first, then acrossphototransmitters 486 and 490 at roughly the same time, and then acrossphototransmitter 488, with the movement generally centered with respectto the phototransmitters 486 and 490. As shown in FIG. 22, an intensitypeak in the measured signal set corresponding to the phototransmitter484 occurs prior to intensity peaks in the measured signal setscorresponding to phototransmitters 486 and 490, and the intensity peaksin the measured signal sets corresponding to phototransmitters 486 and490 occur prior to an intensity peak in the measured signal setcorresponding to the phototransmitter 488. Although not shown, in a casein which a top to bottom slide gesture is performed but where the objectis slightly offset from being centered between phototransmitters 486 and490 such as by being closer to phototransmitter 486, then the graphshown in FIG. 22 would be modified in that the peaks corresponding tocurves 2200 (phototransmitter 484), 2204 (phototransmitter 490), and2206 (phototransmitter 488) would be smaller, and the peak correspondingto curve 2202 (phototransmitter 486) would be bigger.

FIG. 23 is a graph illustrating an analysis for recognizing a side toside slide gesture (also denoted here as a horizontal slide gesture) ofan object from a right side to a left side of an electronic device usingsensing assembly 400. In particular, FIG. 23 illustrates a firstintensity curve 2300 representing a measured signal set corresponding tothe phototransmitter 486, a second intensity curve 2302 representing ameasured signal set corresponding to the phototransmitter 490, acalculated third curve 2304 which represents difference intensityvalues, e.g., intensity values corresponding to the rightphototransmitter 486 minus intensity values corresponding to the leftphototransmitter 490 at respective time periods, and a calculated fourthcurve 2306 which represents average intensity values, e.g., intensityvalues corresponding to an average of intensity values corresponding tothe phototransmitter 486 and the phototransmitter 490 at respective timeperiods.

If the object moves from the right to the left during the slide gesture,then the calculated difference values will first be positive and thenwill be negative, as shown by curve 2304. If an object moves from theleft to the right during the slide gesture, then the calculateddifference values will first be negative and then will be positive. Thuscomputation and analysis of difference values can provide informationregarding the presence and direction of a slide gesture. In some cases,a gesture detection routine can calculate a first difference curverepresenting intensity values corresponding to the rightphototransmitter 486 minus intensity values corresponding to the leftphototransmitter 490, and can also calculate a second difference curverepresenting intensity values corresponding to the left phototransmitter490 minus intensity values corresponding to the right phototransmitter486. A positive signal followed by a negative signal in the firstdifference curve determines that a slide gesture occurred from right toleft, and a positive signal followed by a negative signal in the seconddifference curve determines that a slide gesture occurred from left toright.

The magnitude of the difference signal is dependent on how close theobject is to the sensing assembly when the gesture occurs. In oneembodiment, a corresponding detect threshold 2308 is selected and usedto determine if the difference signal has gone positive an appropriateamount, and a recognize threshold 2310 is selected and used to determinethat the gesture has occurred when the signal goes negative anappropriate amount. These thresholds can provide additional assurancethat a slide gesture has indeed occurred.

Additionally, a slide gesture detection routine can also utilize theaverage intensity values (denoted by curve 2306) of the measured signalsets corresponding to the outlying phototransmitters 486 and 490 and seta clearing threshold 2312 such as shown on curve 2306 with respect tothese average intensity values. If the calculated average intensitysignal falls below this clearing threshold prior to when recognition ofthe gesture has occurred, then the routine is reset and the start of anew gesture is sought.

The slide gesture detection routine can also determine approximate xylocations of the object at different times. For example, referring toFIG. 21, at a time A, the object performing the gesture is generallyabove phototransmitter 486, at a time B, the object is generally abovephototransmitters 484 and 488, and at a time C, the object is generallyabove phototransmitter 490. Various other locations can also bedetermined using interpolation.

A gesture detection routine similar to that described above with respectto FIG. 23 can be employed to detect a top to bottom gesture instead ofa side to side gesture. Further, a similar analysis can apply to thedetermination of a slide gesture in another direction, such as onegenerally along an x=y line.

The electronic device can be operated such that gesture detectionroutines for detection of both vertical (top to bottom or bottom to top)slide gestures and horizontal (side to side) slide gestures operatesimultaneously. In such a case, the predetermined detect and recognizethresholds corresponding to each type of slide gesture can be increasedover that when a single gesture detection routine is operating.

More complex routines can also be employed in order to distinguishbetween slide gestures in the different directions, e.g., to distinguishbetween vertical (top to bottom or bottom to top) slide gestures andhorizontal (right to left or left to right) slide gestures. These can behelpful especially when a slide is performed in one direction, butconflicting signals are also produced that tend to indicate that a slidein another direction has also been performed. For example, this canoccur when a hand or thumb is the object and parts of the wrist or handextend into the active sensing volume and affect the measured signalsets. In order to better distinguish between horizontal and verticalslides, it is recognized that a slope of a difference intensity valuesset over time corresponding to an intended slide direction at a zerocrossing point is greater than a slope of a difference intensity valuesset corresponding to an unintended slide direction.

Specifically, referring to FIG. 24, first vertical difference intensityvalues shown as curve 2400 are calculated with respect to the verticallyaligned phototransmitters (e.g. phototransmitters 484 and 488) andsecond horizontal difference intensity values shown as curve 2402 arecalculated with respect to the horizontally aligned phototransmitters(e.g., phototransmitters 486 and 490). A first slope of the firstdifference intensity values set is calculated at a zero crossing point2403, and a second slope of the second difference intensity set is alsocalculated. Calculation of the first slope can be achieved by takingthree values behind the zero crossing point and one value in front, andcalculating a difference between a maximum and a minimum of thesevalues. In a similar fashion, a second slope corresponding to the seconddifference intensity values can also be determined. If the first slopeis greater than the second slope such as is the case in FIG. 24, then avertical slide gesture is determined to have occurred, while if thesecond slope is greater than the first slope, then a horizontal slidegesture is determined to have occurred.

Various other ways to determine whether an intended gesture has occurredin a horizontal or vertical direction can also be employed, includingcalculating both vertical and horizontal average intensity signal sets,denoted by respective curves 2404 and 2406, and determining whether alargest average value corresponds to either the vertical or horizontalsignal set, with the largest average value indicating that the intendedgesture has occurred in the corresponding vertical or horizontaldirection. Another method involves determining a largest intensity valuecorresponding to one of the phototransmitters at a detection threshold,from which a starting point of a gesture can be inferred. Still anothermethod examines the magnitude of a difference between a positive peakand a negative peak as between horizontal and vertical average signals.

FIG. 25 is an exemplary graph of a curve 2500 representing a measuredsignal set corresponding to a phototransmitter such as phototransmitter486 of sensing assembly 400, wherein a horizontal slide gesture isperformed by a hand in a peace sign configuration (with fingers pointingin a general y direction). In this case, the hand configuration can bedetected by determining the presence of two adjoining peaks in one ormore measured signal sets. As described previously, the timing of thesetwo adjoining peaks as compared to timing of corresponding peaks of oneor more of the other different phototransmitters (such asphototransmitter 490) provides information regarding the direction ofthe slide gesture.

FIG. 26 is an exemplary graph of curves 2600, 2602, 2604, and 2606 whichrepresent measured signal sets corresponding to phototransmitters 484,486, 488, and 490 for a hover gesture which is a pause in movement for apredetermined time period, and which is performed for example as anobject such as an open hand moves from a position generally centeredabove the sensing assembly 400 to a position closer to the sensingassembly and then stays there for a predefined period of time. As shown,curves 2500, 2502, 2504, and 2506 indicate a hover gesture by acorresponding leveling out, where the intensity remains unchanged forthe predetermined amount of time, such as several seconds, for each ofthe measured signal sets. A corresponding distance of the hover gesturefrom the sensing assembly can be determined as described above.

FIG. 27 is an exemplary graph of curves 2700, 2702, 2704, and 2706 whichrepresent measured signal sets corresponding to respectivephototransmitters 490, 484, 488, and 486 for a tilt gesture. In thiscase the tilt gesture is a rotation of an object (such as a open handsituated above the sensing assembly and aligned with fingers pointing ina +y direction) about an axis generally parallel to the y-axis,beginning from a tilted left orientation, rotating through anorientation of the hand generally perpendicular to the mobile device,and then rotating to a tilted right orientation. As shown, an intensitypeak corresponding to phototransmitter 490 has a maximum magnitude whichis greater than the others during the tilted left orientation (timeframe 2708), and the intensity peak corresponding to phototransmitter486 has a magnitude which is less than the others during the tilted leftorientation (time frame 2708). As the hand is moved to an orientationgenerally perpendicular to the mobile device, all of thephototransmitters have generally similar intensity values (time frame2710). During the tilted right orientation (time frame 2712), anintensity peak corresponding to the phototransmitter 486 is greater thanthe others, and an intensity peak corresponding to phototransmitter 490is less than the others. By recognized such patterns in the measuredsignal sets, a tilt gesture can be detected.

With respect to other predefined gestures, including for example a divegesture, or other hand configurations, these other gestures can bedetected by using similar techniques to those described above, namely bydetecting certain patterns or features which have been identified withrespect to corresponding measured signal sets, such as the timing ofintensity peaks in one set with respect to intensity peaks in one ormore of the other sets.

The use of two or more consecutive gestures and detection thereof canprovide additional control possibilities for the electronic device. Manydifferent consecutive gesture sets are possible, which can include thesame or different gestures, and many different operations can beassociated with these different sets. In general, detection ofconsecutive gestures employs the same or similar techniques to thosediscussed above. Note that consecutive gestures are not equivalent to acombination gesture. A combination gesture will not have all signal setsmeasured as near-zero at any time during the gesture. If all signal setsare measured as near-zero, this indicates that no gesture is currentlyoccurring, and thus this lull separates consecutive gestures.

A series of consecutive gestures can be advantageous in order to providemultiple step control of an electronic device. For example, theelectronic device can be operable such that one or more first gesturescan be performed to locate an item, and a second gesture can beperformed to select or launch the item. Specifically, one or moreconsecutive slide gestures can enable a user to scroll within a documentor between a plurality of files when only a portion of the document orfiles can be displayed on a display screen at once. When the userlocates a particular desired portion of the document or a desired file,a hover gesture can be performed in order to select or launch thatcorresponding portion or file.

Another example of a series of consecutive gestures is illustrated inFIGS. 28-31. In particular, these show an object 2800 which movesrelative to an electronic device, such as a mobile device 2802. Mobiledevice 2802 includes a sensing assembly such as sensing assembly 400 ofFIG. 4. As illustrated, a push gesture can first be performed, asindicated by arrow 2804 in FIG. 28, followed by a tilt gesture such as arotation of the object 2800 about an axis parallel to the x-axis, asindicated by arrows 2902, 2904 in FIG. 29. Subsequently, a slide gesturecan be performed, as indicated by arrow 3000 in FIG. 30, with theresultant position and orientation of the object 2800 as shown in FIG.31. Assuming that the position of the object 2800 is initially linked tocontrol a position of a cursor on a display screen of the mobile device2802, this series of gestures can be used for example to first identifya specific item in a stack of items using the push gesture, then selectthe identified item using the tilt gesture, and slide the selected itemto a different area on the display screen using the slide gesture. Ifthese consecutive gestures were performed one after another without anyremoval of the object 2800 between each basic gesture, then they wouldbecome a single combination gesture.

The electronic device can also employ consecutive gesture setsadvantageously so that an identified first gesture operates to determinea particular parameter which can be used in conjunction with a secondgesture to control the electronic device in a variable manner. Forexample, a first hover gesture can be performed, and a correspondingdistance of the object above the sensing assembly during the hovergesture can be determined using a Z distance determination routine asdescribed above. Then one or more slide gestures can be performed tocontrol a scrolling function of the electronic device, wherein ascrolling rate is controlled by the determined distance. In this manner,a hover gesture occurring at a distance of four inches from theelectronic device can result in a scroll rate that is different from ascroll rate resulting from a hover gesture occurring at a distance ofone inch from the electronic device. For example, a complete side toside slide gesture occurring one inch from the electronic device cancorrespond to a scroll rate of one image at a time, while a completeside to side slide gesture occurring three inches from the electronicdevice can correspond to a scroll rate of three images at a time. Aspeed of the performed slide gesture, as calculated by the processor,can also be correlated with a scroll rate of images.

In some cases, a hover gesture is not required, and consecutive slidegestures can control respective scroll rates, with a correspondingdistance of the slide gesture being determined directly and controllingthe respective scroll rate and direction. For example, the time at whichthe middle phototransmitters (phototransmitters 484 and 488 of sensingassembly 400) reach corresponding maximum intensities can be determinedand at that point a z distance determination can be performed in orderto calculate a corresponding z distance. In one embodiment, a larger zdistance is associated with a faster scroll rate, and a smaller zdistance is associated with a slower scroll rate. In another embodiment,a larger z distance is associated with a slower scroll rate, and asmaller z distance is associated with a faster scroll rate.

Other methods for interpreting gestures can be implemented such thatperformance of two consecutive gestures (or gesture sets) of the sametype can result in different control functions being performed. In sucha case, a first occurrence of a specific gesture type can be associatedwith a first mode of operation and a subsequent second occurrence ofthat gesture type can be associated with a second mode of operation.Identification of the first gesture can act to trigger the second modeof operation, while in some cases, identification of an intermediategesture can act to trigger the second mode of operation, or operation ofthe electronic device running a different application can trigger thesecond mode of operation. For example, if instead of operating in aphone call mode, the mobile device described above can be insteadrunning an application for viewing images in an image gallery, in whichcase an identified first gesture (e.g., a first slide or hover gesture)can operate to unlock a different, second mode of operation. In thiscase, when another gesture (e.g., a second slide gesture) issubsequently performed and identified, the identified second gesture islinked to another control function such as a zoom function with respectto a selected image, that is, the second slide gesture controls arespective zooming in or zooming out operation. In such a case, a zoomsetting can correspond to an xy location of the object during the slidegesture.

In some cases, an electronic device can be operable to detectconsecutive gestures (which can be the same type of gesture) using afirst mode of operation associated with a first gesture, and a secondmode of operation associated with a second gesture, wherein the firstgesture acts to “unlock” the second mode of operation because the firstmode of operation links an identification of the first gesture to anactivation of the second mode of operation. In other words, when a firstgesture is identified (the occurrence of the gesture and its type aredetermined from an analysis of the measured signals), the second mode ofoperation is activated, so that an identification of a second gesture islinked to a control function of the mobile device. This operation isadvantageous in that unintended movements of an object near theelectronic device can be ignored until the first gesture of a specifictype is detected. For example, a mobile device can be running a phonecall application which is associated with a first gesture detectionmode. Activation of the phone call application initializes a detectionroutine, and a subsequent identified gesture, for example an identifiedfirst slide gesture or an identified hover gesture, then acts toinitiate a second mode of operation, wherein identification of a secondgesture, such as an identified second slide gesture, is mapped to avolume control function to control the volume of an audio device. Inthis case, the two consecutive gestures can be the same but control thedevice in different ways.

Further, a time limit can be imposed during which the two gestures mustbe completed. Requiring that two consecutive gestures be performedwithin a set time frame in order to cause a change in volume acts toprevent random motion, such as of the user's hand or face, frominadvertently causing a change in volume.

Identification of a first gesture of a consecutive gesture set can alsobe used to control or set which one of two or more different controloperations is be associated with a second detected gesture. Inparticular, the electronic device can be programmed such that a detectedfirst gesture of a first type (i.e., one of the four types of basicgestures) acts to associate a first control operation to a subsequentfirst gesture of a second type, and a second gesture of the first typeacts to set a second control operation corresponding to a subsequentsecond gesture of the second type. For example, a detected first hovergesture can act to control the electronic device such that a subsequentdetected push or pull gesture acts to control a volume of a speaker ofthe electronic device. A detected second hover gesture can act tocontrol the electronic device such that a subsequent second detectedpush or pull gesture acts to control a zoom function. Thus, subsequenthover gestures (e.g., gestures of the first type) can act to togglebetween the different control operations that are adjusted by push/pullgestures (e.g., gestures of the second type).

Additionally, in certain situations other techniques can be employed inorder to aid in the interpretation of user intent with respect toconsecutive gestures. For example, consecutive slide gestures cancontrol corresponding scrolling operations of items displayed on adisplay screen of an electronic device, such as to scroll between one ormore displayed photographs or to turn pages of an e-book. Typically insuch a case, a user will repeatedly perform the same slide gesture toadvance through displayed items in the same desired scroll direction andwithout changing the desired scroll direction.

In this case, detection of a slide gesture such as in a +x directionoperates to control the scrolling of items in a corresponding directionon the display screen, and detecting two or more consecutive slidegestures in that same direction to control consecutive scroll operationscan be problematic because the object (e.g., hand or finger) must bemoved from an ending position of a first desired slide gesture to thebeginning position of a second desired slide gesture. This movement ofthe object in the negative x direction back to the beginning point maybe interpreted as a slide gesture in that negative x direction, and actto control a scroll function in the corresponding backwards direction,which may not be desired.

In such a case, it can be advantageous for a user to perform the firstslide gesture such that the object (such as a hand or thumb or finger)is moved in the desired direction at a closer distance to the sensingassembly than when the object is moved back in the opposite direction.If the magnitude of a detection threshold is set high enough, it can bepossible that the second gesture will not be detected. However, variousother ways exist for “blanking out” or ignoring a reverse directionslide gesture (e.g. a second slide gesture which occurs in an oppositedirection from the direction associated with an identified first slidegesture).

As further described below, several different types of blanking routinescan be used, including a “soft” blanking routine which ignores a secondgesture based on determined characteristics of the first gesture and/orsecond gesture, and a “hard”blanking routine which implements apredetermined time frame during which any gesture that may occursubsequent to a first identified gesture is ignored.

A threshold based soft blanking routine can operate as follows. After anidentification of a first gesture, the magnitude (absolute value) of arecognition threshold, a detection threshold, and/or a clearancethreshold (as described above and shown in FIG. 23) applicable todetection of a second gesture can be changed from a correspondingthreshold(s) applicable to the first gesture. The soft blanking routinecan therefore act to ignore a second slide gesture, such that a nullfunction is associated with that second slide gesture. As describedbelow, the threshold can be changed according to certain calculatedparameters of the first detected gesture, such as a calculated distanceof the first slide gesture from the sensing assembly.

FIG. 32 helps to illustrate the operation of an exemplary thresholdbased soft blanking routine for consecutive slide gestures, wherein anidentified slide gesture is linked to a corresponding function, such asa scrolling function, in a first mode of operation. In particular, FIG.32 illustrates the case where a first slide gesture is performed in aright to left direction over an electronic device with sensing assembly400 during a first time period 3200, and a second slide gesture is thenperformed in a left to right direction during a second time period 3202.Note that FIG. 32 clearly shows two consecutive gestures, and not asingle combination gesture, because of the time period where all thesignal sets go to near-zero. In particular, FIG. 32 illustrates a curve3204 which represents calculated difference intensity values, in thiscase intensity values corresponding to the right phototransmitter 486minus intensity values corresponding to the left phototransmitter 490 atrespective time periods. FIG. 32 also illustrates a curve 3206 whichrepresents calculated average intensity values, here an average of theintensity values corresponding to the phototransmitter 486 and thephototransmitter 490 at respective time periods. A curve 3208 representsmaximum intensity values corresponding to the maximum of eitherphototransmitters 484 or 488.

Curve 3208 provides a height or distance estimate (the lower theintensity, the greater the z-axis height) of the gesture, and a maximumvalue 3210 is determined over a period of time during which the firstgesture is occurring, such as from when the difference signalcorresponding to the right to left gesture crosses a first detectionthreshold 3212 to when the difference signal crosses a first recognitionthreshold 3214 (as described above with respect to FIG. 23). After theright to left gesture is detected, a new second recognition threshold3216 can be calculated which is applicable to the next gesture and whichmakes detection less sensitive for a predetermined period of time. Inthis manner, a subsequent left to right gesture will not be identified(will be ignored), or is less likely to be identified (although it isdesirable that a true left to right gesture be detected). For example,the magnitude (absolute value) of the recognition threshold 3216 can beincreased by a predetermined amount over the recognition threshold 3214,and for a predetermined amount of time. In one case, a detectionthreshold is set to be 0.9 times the maximum value of signal 3208 for atime period of 1 second (which corresponds to 60 samples at a samplingrate of 16 msec/sample).

The calculated parameters for setting the detection threshold can bedifferent with respect to on-glass slide gestures and off-glassgestures, where on-glass gestures are those performed in a non-touchlessmanner such as directly on the glass of a display screen of anelectronic device. Preferably, an on-glass soft blanking routine isperformed using a low power setting for the phototransmitter(s) becausemore of the light emitted from a phototransmitter is reflected backusing an on-glass gesture as compared to an off-glass gesture. In such acase, determining a difference in distances between an object movingon-glass across the electronic device and the object moving slightlyoff-glass across the electronic device is difficult because both casesproduce similar high intensity values and accurate distance resolutionin not achievable.

With respect to off glass gestures, here a z distance calculationroutine can better determine distances of the slide gesture above thesensing assembly, and a soft-blanking threshold corresponding to thesecond gesture can be changed accordingly in response to a calculated zdistance of the first gesture.

A threshold based soft blanking routine can operate as follows.Basically, a calculated duration (or speed) of a second slide gesture iscompared to a setpoint duration (or speed) in order to distinguishbetween a first slide gesture in a first direction and the second slidegesture in a second and opposite direction, and to ignore the secondslide gesture depending on the results of the comparison. The setpointduration can be a default or user specified amount of time, or can be acalculated duration corresponding to the first slide gesture. FIG. 33helps to illustrate the operation of an exemplary threshold based softblanking routine for consecutive slide gestures, wherein an identifiedslide gesture is linked to a corresponding function, such as a scrollingfunction, in a first mode of operation. In particular, FIG. 33 (likeFIG. 32) illustrates the case where a first slide gesture is performedin a right to left direction over an electronic device with sensingassembly 400, and a second slide gesture is then performed in a left toright direction. In this case, the first gesture is performed moreslowly than the second gesture, and both are performed at approximatelythe same distance from the sensing assembly 400, although it is alsopossible that the two gestures be performed at different distances. Inthe latter case, blanking can be achieved and at the same time it ispossible for the z-distance of a slide gesture to be used to set someparameter such as a scroll rate of a corresponding electronic devicefunction as described above.

Similar to curve 3204 of FIG. 32, a calculated curve 3300 is illustratedin FIG. 33, which represents calculated difference intensity values, forexample intensity values corresponding to the right phototransmitter 486minus intensity values corresponding to the left phototransmitter 490 atrespective time periods. In this case, the first gesture can beidentified by determining that the difference intensity values of curve3300 cross both a recognition threshold 3302 and a detection threshold3304 (as described above with respect to FIG. 23). A duration 3308 ofthe first gesture can be calculated by determining when these crossingsoccur. A duration 3306 of the second gesture can also be calculated. Inother cases, a duration of a gesture can be defined in a somewhatdifferent manner using threshold levels which can be different than therecognition and duration thresholds shown.

In one case, if the duration 3306 is determined to be smaller than asetpoint duration, which can be the calculated duration 3308, then thesecond gesture will be ignored. Thus, no corresponding scroll actionwill occur when a second gesture is performed faster than the firstgesture (although operation can also be defined in the opposite manner,such that a second slower gesture following a first faster gesture willbe ignored).

Additionally, a predetermined time frame, such as two seconds, can beimposed during which both the first gesture and the second gesture haveto be performed, in order for the blanking mode to be active. If thepredetermined time frame limitation is not met, then the blanking modewill be inactive and the two gestures will essentially be treatedindependently.

A hard blanking routine simply implements a predetermined time duration3310 such as shown in FIG. 33 which occurs subsequent to identificationof a first gesture and during which the second gesture (or any othergesture) is ignored. Following an identified first slide gesture, it isassumed that the second gesture occurs during the time duration 3310such that measured signals are not acquired or are simply ignored. Thishard blanking routine is time based and is not level or threshold based.A hard blanking routine can be less attractive to users because a userhas to wait for the predetermined time duration to pass before thedevice can recognize a subsequent slide gesture. In some cases, thepredetermined time duration can be user specified via a user interface,such that a user can select or specify a desired time duration forblanking. It can also be possible for a user to select no blanking andin such a case, the user would have to make an effort to return theobject to the side to avoid crossing and/or triggering the transmittedinfrared light beams in a manner so that the return gesture is notrecognized.

It is specifically intended that the present invention not be limited tothe embodiments and illustrations contained herein, but include modifiedforms of those embodiments including portions of the embodiments andcombinations of elements of different embodiments as come within thescope of the following claims.

We claim:
 1. A method for interpreting at least two consecutivegestures, wherein each gesture is a specified pattern of movement of anexternal object relative to an electronic device, the method comprising:providing as part of the electronic device a sensing assembly includingat least one photoreceiver and a plurality of phototransmitters, whereineach phototransmitter is positioned to emit infrared light away from theelectronic device about a corresponding central transmission axis,wherein each central transmission axis is oriented in a differentangular direction with respect to the others; controlling emission ofinfrared light by each of the phototransmitters during each of aplurality of time periods; for each of the plurality ofphototransmitters and for each of the plurality of time periods,generating a corresponding measured signal indicative of a respectiveamount of infrared light which originated from that phototransmitterduring that time period and was reflected by the external object priorto being received by a photoreceiver; evaluating the measured signals toidentify a first gesture; controlling the electronic device in responseto the identification of the first gesture according to a first mode ofoperation of the electronic device; determining a parameter of a secondgesture; and controlling the electronic device in response to thedetermined parameter of the second gesture according to a second mode ofoperation.
 2. The method of claim 1, wherein the first mode of operationlinks the identification of the first gesture to a mode change functionthat activates the second mode of operation, and wherein each of thegestures is selected from the group consisting of a slide gesture, apush gesture, a pull gesture, a tilt gesture, a hover gesture, and acombination gesture formed from at least two of the slide gesture, thepush gesture, the pull gesture, the tilt gesture, and the hover gesture.3. The method of claim 1, wherein identification of an intermediategesture subsequent to occurrence of the first gesture and prior tooccurrence of the second gesture triggers a change between the firstmode of operation and the second mode of operation.
 4. The method ofclaim 1, wherein determining a parameter of the second gesture includesassuming an occurrence of the second gesture in response to theidentification of the first gesture, and the second mode of operationlasts for a predetermined duration and links an assumed occurrence ofthe second gesture to a null function.
 5. The method of claim 4, whereinthe first gesture is a slide gesture in a first direction and the secondgesture is a slide gesture in a second direction that is opposite to thefirst direction, and the first mode of operation links an identificationof a slide gesture to a scrolling, a panning, or a zooming function. 6.The method of claim 5, wherein after the predetermined duration haspassed, the first mode of operation is again activated such that asubsequent identified slide gesture controls a scrolling, a panning, ora zooming function.
 7. The method of claim 4, wherein the predeterminedduration is selectable by a user.
 8. The method of claim 1, whereinevaluating the measured signals also includes: determining a firstidentification threshold, wherein determining a parameter of the secondgesture includes determining whether a calculated measured signal setcorresponding to the second gesture includes a signal having a magnitudewhich is greater than the first identification threshold, and if so,identifying the second gesture, and if not, ignoring the second gesture,and wherein the second mode of operation links an ignored second gestureto a null function and links an identification of the second gesture toa function that is the same as the function associated with the firstmode of operation.
 9. The method of claim 8, wherein the first gestureis a slide gesture in a first direction and the second gesture is aslide gesture in a reverse direction, and the first mode of operationlinks an identification of a slide gesture to a scrolling, a panning ora zooming function.
 10. The method of claim 1, wherein the determining aparameter of the second gesture includes: calculating whether the secondgesture is of greater duration than a setpoint duration, and wherein thesecond mode of operation links a determination that the second gestureis of shorter duration than the setpoint duration to a null function,and links a determination that the second gesture is of greater durationthat the setpoint duration to a function that is the same as a functionassociated with the first mode of operation.
 11. The method of claim 10,wherein the first gesture is a slide gesture in a first direction andthe second gesture is a slide gesture in a reverse direction, and thefirst mode of operation links an identification of a slide gesture to ascrolling function.
 12. The method of claim 10, wherein evaluating themeasured signals to identify the first gesture also includes calculatinga duration of the first gesture, and using the duration of the firstgesture as the setpoint duration.
 13. A method for detecting two or moreconsecutive specified patterns of movement of an external objectrelative to an electronic device, the method comprising: providing aspart of the electronic device a sensing assembly including at least onephototransmitter and a plurality of photoreceivers, wherein thephototransmitter is positioned to emit infrared light away from theelectronic device about a central transmission axis, and eachphotoreceiver is positioned to receive infrared light about acorresponding central receiving axis, wherein each central receivingaxis is oriented in a different direction with respect to the others;controlling emission of infrared light by the phototransmitter duringeach of a plurality of sequential time periods; for each of theplurality of photoreceivers and for each of the plurality of sequentialtime periods, generating a corresponding measured signal indicative of arespective amount of infrared light which originated from thephototransmitter during the corresponding time period and was reflectedby the external object prior to being received by the correspondingphotoreceiver; and evaluating the measured signals to identify a firstgesture, wherein the first gesture includes a first of the two or moreconsecutive specified patterns of movement; controlling the electronicdevice in response to the identification of the first gesture accordingto a first mode of operation of the electronic device; determining aparameter of a second gesture; and controlling the electronic device inresponse to the determined parameter of the second gesture according toa second mode of operation.
 14. The method of claim 13, wherein thefirst mode of operation links the identification of the first gesture toa mode change function that activates the second mode of operation. 15.The method of claim 13, wherein identification of an intermediategesture subsequent to occurrence of the first gesture and prior tooccurrence of the second gesture triggers a change between the firstmode of operation and the second mode of operation.
 16. The method ofclaim 13, wherein determining a parameter of the second gestureincluding assuming an occurrence of the second gesture in response tothe identification of the first gesture, and the second mode ofoperation lasts for a predetermined duration and links an assumedoccurrence of the second gesture to a null function.
 17. The method ofclaim 13, wherein evaluating the measured signals also includesdetermining a first identification threshold, wherein determining aparameter of the second gesture includes determining whether acalculated measured signal set corresponding to the second gestureincludes a signal having a magnitude which is greater than the firstidentification threshold, and if so, identifying the second gesture, andif not, ignoring the second gesture, and wherein the second mode ofoperation links an ignored second gesture to a null function and linksan identification of a second gesture to a function that is the same asthat associated with the first mode of operation.
 18. The method ofclaim 13, wherein the determining a parameter of the second gestureincludes: calculating whether the second gesture is of greater durationthan a setpoint duration, and wherein the second mode of operation linksa determination that the second gesture is of shorter duration than thesetpoint duration to a null function, and links a determination that thesecond gesture is of greater duration that the setpoint duration to afunction that is the same as a function associated with the first modeof operation.
 19. A method for interpreting at least two consecutivegestures, wherein each gesture is a specified pattern of movement of anexternal object relative to an electronic device, the method comprising:providing as part of the electronic device a sensing assembly includingat least one photoreceiver and a plurality of phototransmitters, whereineach phototransmitter is positioned to emit infrared light away from theelectronic device about a corresponding central transmission axis,wherein each central transmission axis is oriented in a differentdirection with respect to the others; controlling emission of infraredlight by each of the phototransmitters during each of a plurality oftime periods; for each of the plurality of phototransmitters and foreach of the plurality of time periods, generating a correspondingmeasured signal indicative of a respective amount of infrared lightwhich originated from that phototransmitter during that time period andwas reflected by the external object prior to being received by aphotoreceiver; evaluating the measured signals to identify a firstgesture; controlling the electronic device in response to theidentification of the first gesture according to a first mode ofoperation of the electronic device; determining a parameter of a secondgesture, wherein the first gesture and the second gesture are the sametype of gesture and the first mode of operation and the second mode ofoperation are different; and controlling the electronic device inresponse to the determined parameter of the second gesture according toa second mode of operation.
 20. The method of claim 19, wherein thefirst gesture and the second gesture are performed in oppositedirections from each other.